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Executive Summary 


At the turn of the century, the National Science Board (NSB) noted that “international boundaries 
have hecome considerably less important in structuring the conduct of research and development” in 
science and engineering fields (S&E) (NSB, 2001). NSB called for increased U.S. government 
commitment to international S&E research and education, and acknowledged the National Science 
Foundation’s (NSE) leadership role in international S&E research and education activities (NSB, 
2001). 

NSF’s Partnerships in International Research and Education (PIRE) program, which funded its first 
cohort of projects in 2005, reflects NSF’s commitment to international partnerships that can help 
address critical S&E questions. PIRE funds projects across a broad array of scientific disciplines, in 
an effort to catalyze long-term, sustainable international partnerships for collaborative research and 
education that will prepare a cadre of students and early-career researchers for strong leadership and 
engagement in global science and engineering. Projects support intellectually substantive 
collaborations between U.S. and foreign researchers in which the international partnership is essential 
to the research effort. 

To better understand the results of its investment in PIRE, NSE contracted with Abt Associates to 
conduct an evaluation of PIRE. We designed the evaluation to examine the value added when U.S. 
researchers engage foreign collaborators in shaping and implementing a research agenda through 
PIRE, specifically to: examine the research produced by projects funded by the PIRE program; 
measure the research and career outcomes for PIRE participants; explore the perceived effects of 
PIRE on U.S. and foreign institutions; and document the contributions of PIRE to addressing societal 
challenges. 

The evaluation used existing and primary data sources and descriptive, comparative, and quasi- 
experimental methods to provide information on outcomes in four key areas: 

• Research outcomes, including the quantity and quality (i.e., citation impact) of projects’ 
publications, and the percentage of foreign coauthors of these publications; 

• Participant experiences during and after the project, in particular the nature of project-related 
activities conducted outside the U.S., collaborations between U.S. and foreign participants both 
during and subsequent to project participation, and perceived benefits and challenges of PIRE; 

• Role of U.S. institutional policies and practices surrounding international faculty and student 
engagement and how these policies and practices affected — or were affected by — the PIRE 
project; and 

• The potential contributions of PIRE to global societal challenges that may require 
international cooperation and coordination. 

A central component of the evaluation methodology was the construction of a matched comparison 
group of NSE-funded projects that were not required to include an international collaboration and that 
were matched to PIRE projects along a set of key criteria such as award amount and duration. 

The evaluation found evidence that PIRE is furthering its objectives: to promote opportunities for 
U.S. scientists and engineers to engage in international collaborations that enhance research 
excellence; to provide international research and educational experiences for U.S. students and 
faculty that will prepare the U.S. science and engineering workforce for global engagement; to 
strengthen the capacity of U.S. researchers and institutions to build and sustain international 
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partnerships; and to support partnerships that promote excellence in science, engineering and 
education. 


Specific Findings 

Research Outcomes 

• Across PIRE journal articles the mean field normalized citation impact (NCI) was 1.8 and the 
journal NCI was 1.3, reflecting a higher than average impact of these articles relative to the 
impact of an average journal article (the average field and journal NCI =1). 

• There were no statistically significant differences between PIRE and the comparison group in the 
quantity or quality of research produced. However, PIRE publications had a significantly higher 
proportion of foreign contributions per journal article measured both as the percentage of foreign 
institutions per article and the percentage of foreign authors per article. 

• At the participant level, there were no significant differences in annual number of post-onset 
publications between PIRE and comparison Principal Investigators (Pis). However, among 
postdocs, PIRE participants had, on average, a larger number of post-onset annual publications 
and these publications had a larger impact. PIRE graduate students also had a higher average 
number of annual post-award publications than their comparison group peers. 

• PIRE Pis’ post -onset publications had, on average, a statistically significant higher percentage of 
non-U.S. institutional affiliations among authors than comparison Pis’ post-onset publications. 

Participant Experiences 

• A greater proportion of Pis, postdoctoral researchers (postdocs), and graduate students in PIRE 
projects traveled outside the U.S. than participants in comparison projects. Eor each group, the 
percent was well over three-fifths among PIRE participants and less than one-quarter among 
participants in comparison projects. Eurther, PIRE postdocs and graduate students were more 
likely than their counterparts in comparison projects to have traveled abroad to conduct research. 

• Collaboration with individuals within the U.S. was commonly reported by both PIRE and 
comparison Pis, postdocs and graduate students, however, PIRE participants were more likely to 
collaborate with individuals based primarily outside the U.S. 

• PIRE Pis and graduate students were more likely to report maintaining these international 
collaborations than comparison project participants, and commonly these relationships continued 
to focus on research-related exchanges. 

• PIRE postdocs, graduate students, and undergraduates would recommend their peers engage in 
international research and education because of the benefits associated with working with and 
learning from international collaborators, cultural and social exchange; enhanced professional 
networking, and skills and knowledge developed as a researcher. Challenges were not commonly 
mentioned by respondents. 

• Undergraduates who traveled outside the U.S. for PIRE-related research were more likely to have 
collaborated with foreign faculty, senior- or postdoctoral-level researchers or graduate students 
rather than with foreign undergraduate students. 

• Among undergraduate respondents who traveled outside the U.S. for PIRE-related research, the 
majority reported that their participation in the project yielded a range of personal benefits. Most 
commonly reported among these was an understanding of how scientists work on real problems 
(81 percent), followed by tolerance for obstacles faced in the research process (71 percent), self- 
confidence in their field of study (69 percent), and clarification of a career path (67 percent). 
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• Foreign senior investigators (FIs) were less likely than U.S. -based investigators to have traveled 
internationally during their participation in the PIRE project (48 percent). Among those who 
traveled abroad, the majority traveled to the U.S. (78 percent). 

• Among most FIs, PIRE was not the first international research collaboration with a U.S. -based 
partner; 82 percent reported working with a U.S. -based individual on research prior to their 
participation in the PIRE project. 

Institutional Context 

• Institutional representatives reported that their institutions support international research and 
education initiatives through a variety of mechanisms, and the majority indicated that policies and 
practices mostly facilitated graduate and undergraduate participation in international 
opportunities. 

• Institutional representatives found that PIRE aligned well with existing institutional goals for 
internationalization. 

A greater number of representatives described institutional factors that support undergraduate and 
graduate student involvement in education and research activities abroad than the number that 
described barriers to such involvement. However, more representatives mentioned barriers to 
undergraduate than graduate student participation in international opportunities. 

• The majority of administrators responded that PIRE had not resulted in any changes in policies or 
practices related to undergraduate or graduate education in science, technology, engineering and 
mathematics (STEM) fields (32 respondents). Among the 14 responses that credited PIRE with 
effecting change, many indicated that faculty, students and administrators had increased 
awareness of the value of international research experiences for students. 

Societal Impacts 

• The three most frequent types of benefits of PIRE described by senior investigators were insights 
about the natural environment (43 percent of PI responses, 24 percent of PI responses); 
technological advances that would benefit society (21 percent of PI responses) and PlRE’s 
promotion of cross-cultural exchange (18 percent of PI responses, 32 percent of PI responses). 
Specific examples included scientific advances in sustainable energy, conservation, 
understanding climate change, and mitigating hazards from disasters. 

• Senior investigators (74 percent of Pis and 78 percent of Pis) noted that their PIRE project has 
advanced knowledge or led to discoveries that might help address global challenges in health, 
environmental and other areas in which solutions appear unlikely to emerge from the efforts of 
one nation acting alone. Asked how their PIRE project could help address these challenges, the 
mechanism that senior investigators most often mentioned was that PIRE projects leveraged the 
expertise, resources, and perspectives of a range of international researchers. 

• Pis and Pis indicated that findings from their PIRE project could inform policy and planning, 
provide tools or technologies applicable to global challenges, and provide improved academic 
models for educating the next generation of researchers. The characteristic of PIRE that lead 
investigators most commonly mentioned as contributing to addressing global challenges is the 
ability to leverage the expertise, resources, and perspectives of a range of international 
researchers. 
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1. Introduction 


At the turn of the eentury, the National Scienee Board (NSB) noted that “international boundaries 
have become considerably less important in structuring the conduct of research and development” in 
science and engineering (S&E) fields (NSB, 2001). NSB called for increased U.S. government 
commitment to international S&E research and education, and acknowledged the National Science 
Foundation’s (NSE) leadership role in international S&E research and education activities (NSB, 
2001). 

The Office of International Science and Engineering (OISE) is NSF’s hub for international research 
opportunities for U.S. scientists and engineers. Operated within OISE, the Partnerships in 
International Research and Education (PIRE) program, which funded its first cohort of projects in 
2005, reflects NSF’s commitment to international partnerships that can help address critical S&E 
questions. Dr. Jon Strauss, a member of the NSB and former Chairman of NSB’s Task Force on 
International Science, cited PIRE in his Congressional testimony as an example of a federal program 
that fosters innovative research and education by creating cross-border collaborations between U.S. 
scientists and engineers and international colleagues (Strauss, 2009). 

PIRE funds projects across a broad array of scientific disciplines, in an effort to catalyze long-term, 
sustainable international partnerships for collaborative research and education that will prepare a 
cadre of students and early-career researchers for strong leadership and engagement in global science 
and engineering. Projects support intellectually substantive collaborations between U.S. and foreign 
researchers in which the international partnership is essential to the research effort. 

To better understand the results of its investment in PIRE, NSE contracted with Abt Associates to 
conduct an evaluation of PIRE. We designed the evaluation to examine the value added when U.S. 
researchers engage foreign collaborators in shaping and implementing a research agenda through 
PIRE, specifically to: 

• Examine the research produced by projects funded by the PIRE program; 

• Measure the research and career outcomes for PIRE participants; 

• Explore the perceived effects of PIRE on U.S. and foreign institutions; and 

• Document the contributions of PIRE to addressing societal challenges. 

1 .1 Overview of the Study 

The evaluation used descriptive and matched comparison analyses to: examine the quantity and 
quality of research produced by the PIRE program and its participants; measure the research and 
career outcomes for PIRE participants; and document how PIRE is perceived as changing the way 
U.S. institutions support international research and educational collaborations. The evaluation was 
designed to answer the following research questions: 

1. What is the quantity and quality of the publications that PIRE projects have produced? 

2. How does the quantity and quality of publications produced by PIRE projects — that are required, 
by definition, to include an international collaboration — compare to the quantity and quality of 
publications produced by similar NSE-funded projects that do not require an international 
collaboration? 
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3. (a) What are the research outcomes for PIRE participants (Pis, postdoctoral researchers and 
graduate students)? 

(h) What are the program experiences and educational or career outcomes of PIRE Pis, 
postdoctoral researchers, and graduate and undergraduate student participants? 

4. (a) How do the research outcomes for PIRE Pis, postdoctoral researchers, and graduate students 
compare to those of similar participants in similar NSE-funded projects that do not require an 
international collahoration? 

(h) How do the program experiences and educational or career outcomes of these PIRE 
participants compare to similar participants in this comparison group of NSE-funded projects? 

5. How do international affairs representatives at PIRE lead Pis’ institutions perceive the effects of 
PIRE on their institutions’ policies and practices for supporting international research and 
educational collahorations? 

6. What are the program experiences of foreign senior investigative partners in PIRE and how do 
they perceive the effects of PIRE on their research and educational practices and those of their 
institutions? 

7. How have PIRE projects contributed to research that may inform global societal challenges? 

To answer these questions, the evaluation relied on extant data as well as data collected via surveys 
designed specifically for this evaluation. 

The 59 PIRE projects awarded as of June 2013 were included in the evaluation; NSE had made 12 
awards in 2005, 20 in 2007, 15 in 2010 and 12 in 2012. These projects range in size from relatively 
small, bi-national consortia (e.g., two U.S. and two non-U.S. institutions in one foreign country) to 
large, multi-national, multi-institutional awards (e.g., a dozen U.S. institutions and 11 non-U.S. 
institutions representing eight foreign nations). Many are multi-disciplinary, combining, for example, 
the expertise of econometricians with researchers in fluid dynamics; and, notably, many feature 
partnerships between academic and industrial or non-profit institutions. The comparison group of 
projects was constructed from non-PIRE, NSE-funded projects in which an international collaboration 
is not required. Project-level characteristics of the PIRE projects were used to identify the comparison 
set of projects. 

1 .2 Context of the Study 

International partnerships in S&E are vital both for understanding critical science and engineering 
problems, and for advancing discoveries in STEM fields. Through international networks of 
scientists, resources can be shared and ideas can be developed, tested, and implemented across 
traditional boundaries (NSB, 2008). Such partnerships can also serve as an important tool in broader 
international diplomacy efforts. Policymakers also note 
that including developing nations in collaborative 
efforts can promote self-sufficiency and encourage 
international participation beyond S&E research alone 
(NSB, 2008). These collaborations are complex. 

Recently, a workshop convened by the U.S. National 
Academy of Sciences explored the role that culture, 
administrative, and legal considerations play in 
international research collaborations (National 
Research Council, 2014). 


Science and engineering partnerships 
can strengthen internationai 
reiationships and. . . promote basic 
scientific vaiues such as accountabiiity, 
meritocracy, transparency, and 
objectivity. 

—NSB, 2008 
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NSB recognized this globalization of S&E research and education, in concert with the associated 
opportunities and challenges for the U.S., in a 2010 publication highlighting globalization trends 
(NSB, 2010). Further, the potential of science policy and science diplomacy to meet international 
challenges was highlighted by participants at a workshop convened by the National Research Council 
to discuss the importance of international science engagement and global science cooperation, and the 
value of providing opportunities and incentives for U.S. researchers to engage in science in an 
international arena (National Research Council, 201 1). 

Engagement in international collaboration is important for U.S. scientists’ continued prominence in 
the global research community and capacity to gain insight into international research advances 
(NSB, 2001). An active role in international collaborations means that the U.S. can expect to reap 
benefits not otherwise realized. Recent research found that when U.S. and U.K. researchers engaged 
in collaborations, the impact of their resulting research significantly increased (as measured by 
citation rates), especially for U.S. corresponding authors (Luo et ah, 201 1). Similarly, a recent 
bibliometric analysis found that papers written by collaborators from multiple countries were cited 
more than those with authors from a single country (Adams, 2013). Additional benefits of 
international collaboration may include increased access to physical resources and funding; additional 
opportunities to benefit from the expertise of collaborators; and access to populations, records, 
historical materials, and circumstances that provide “natural experiments” (Goodnow, 2008). 
International collaborations can facilitate expansion of U.S. markets and promote opportunities for 
international economic exchange (NSB, 2001). Finally, international networks of scientists can more 
easily share resources and develop, test, and implement ideas across traditional boundaries (NSB, 
2008). 

Although policy-makers and funding agencies have emphasized the increasing need for U.S. S&E 
researchers to become globally engaged with their counterparts in other countries, little research has 
systematically examined the effectiveness of programs explicitly designed to foster international 
collaboration. Recent studies have, however, documented that participants in programs designed to 
promote international S&E collaboration report acquisition of knowledge and skills, as well as plans 
to engage in international collaborations in the future. For example, an evaluation of NSF’s 
International Research Experience Program (IREP), a program for undergraduate and graduate 
students in five of NSF’s Science and Technology Centers, found that students who engaged in an 
international research experience reported gaining technical, communication, and language skills; 
developed an appreciation for cultural differences; and felt that their research experience would 
“create opportunities for future international collaboration” (Spencer, 2008).' An evaluation of the 
Research Internships in Science and Engineering (RISE) program reached similar conclusions. RISE 
supports undergraduate students’ summer internships in German higher education institutions. 
Researchers found that the majority of RISE student participants developed a greater understanding of 
German culture, the vast majority reported an increased desire to travel abroad, and about one in four 
alumni reported returning to Germany at some point after the internship, either to visit or to pursue 
work/study opportunities (Institute of International Education, 2009). Finally, an evaluation of NSF’s 
International Research and Education in Engineering (IREE) pilot program also found that program 
participants, especially graduate and postdoctoral students, reported acquiring new research 


* Begun in 2005 by NSF’s Office of Integrative Affairs, the IREP program invited Science and Technology 
Centers (STCs) to apply for funding to (1) provide students at STCs with an international research 
experience; (2) establish or strength an international collaboration that would enhance the STC’s mission; 
and (3) provide students a more international experience at their home STC. From the five STCs selected, 
36 graduate students and 6 undergraduates traveled to 18 countries, for stays lasting approximately 2 to 3 
months. The program ended in 2008. 
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capabilities, and that they planned to continue collaborations with their international counterparts at 
the end of their program experience (Flattau et al., 2009). 

Some recent evaluations of other NSF programs have employed rigorous quasi-experimental design 
to construct appropriate comparison groups to investigate the project-related career outcomes for 
program participants related to a matched comparison using propensity score matching for the IRFP 
(Martinez, Epstein, et al., 2012) and EAPSI (Martinez, Neishi, et al., 2012b) programs. These studies 
found that a large proportion of participants continued their international collaborations beyond the 
fellowship period (for both programs). In comparison to their matched peers who did not receive a 
fellowship, fellows in both programs produced a larger number of publications co-authored by a 
collaborator in a foreign institution, although the difference was larger for the IREP than the EAPSI 
program, which offers a shorter fellowship period (Martinez et al., 2012a; Martinez et al., 2012b). 

The proposed evaluation will further our understanding of the role of programs in fostering 
international research collaborations, and the productivity and influence of such collaborations. 

1.3 PIPE Program 

The PIRE program makes competitive awards for up to five years of funding to U.S. scientists and 
engineers who partner with individuals and institutions outside the U.S. to implement a program of 
research and education. Classes are funded in an alternating two-year, three -year cycle so that two 
classes of awards are active at any given time. The first class of funded projects was 2005, so their 
funding ended in 2010 to 201 1; the second class was funded two years later, and ended in 2012. 
However, there have been some no-cost extensions. The third class was funded in 2010 and the fourth 
in 2012. Across these first four cohorts the program made a total of 59 awards (two of which were 
renewals of prior PIRE awards). Eor the first two PIRE competitions, the maximum funding per 
award was $2.5 million (up to $500,000 per year). These restrictions were removed for the 2010 and 
2012 cohorts; the result was an approximate doubling in the amount of per-award funding across the 
five year grant period. PIRE is one of NSE’s “cross-cutting” programs open to researchers within and 
across any of the disciplines NSE typically funds through its directorates. 

PIRE grantee institutions must provide professional development opportunities for U.S. -based 
researchers, especially those early in their career, including postdoctoral fellows and graduate and 
undergraduate students. A significant aspect of the opportunity for U.S. participants is on-site 
research experience at an international laboratory, whether university-, industry- or government- 
based. PIRE funding can be used to support the costs of U.S. -based participants in the international 
collaboration; foreign partners contribute funding from their own sources, such as NSF’s counterpart 
agencies in other countries. 

Eor PIRE competitions, NSE has limited the number of proposals that each lead institution can 
submit. Starting with the second PIRE competition, NSE has used a multi-stage proposal process, in 
which institutions first submit a preliminary proposal and NSE then selects which of these are invited 
to submit a full proposal. Up to four external reviewers (identified based on the proposed research 
areas) rate each PIRE proposal along a 1 (poor) to 5 (excellent) scale; subsequently, reviewers meet in 
panel to discuss each proposal and recommend to NSE which proposals are “not competitive,” 
“competitive,” or “very competitive.” NSE program officers make the final award recommendations 
(subject to the available budget and approval by NSE management). These award recommendations 
take into account both panel recommendations and other factors such as the geographic and 
disciplinary balance across the portfolio of awards as well as participation by Pis traditionally under- 
represented in science and engineering, and institutions from states that typically receive a lower 
share of NSE funding. 
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The PIRE program has historically received the largest share of NSF’s Office of International Science 
and Engineering (OISE) programming budget: for example, in FY 2011, the PIRE program had a 
budget of approximately $18.7 million, representing 38 percent of OISE’s total budget (NSF, 2012).^ 
The objectives of the PIRE program are to: promote opportunities for U.S. scientists and engineers to 
engage in international collaborations that enhance research excellence; provide international research 
and educational experiences for U.S. students and faculty that will prepare the U.S. science and 
engineering workforce for global engagement; and strengthen the capacity of U.S. researchers and 
institutions to build and sustain international partnerships. 

Notable changes to the program since its inception include changes to scale and scope. In the 2005 
and 2007 PIRE cohorts, budgets were capped at $2.5 million, but beginning with the 2010 cohort, 
these budget limitations were removed (NSF, 2004, 2006, 2007, 2009). In addition, starting with the 
2010 cohort, grantees had to propose a project of sufficient scope that its effects would extend beyond 
an individual Pi’s research group to the participating U.S. institutions by strengthening institutional 
capacity for sustained international engagement. For the 2012 PIRE cohort, NSF tailored its first 
objective (to support international collaborations that enhance research excellence) to focus on 
international partnerships that would support excellence in science, engineering and education to 
“inform the societal actions needed for environmental and economic sustainability and sustainable 
human well-being” (NSF, 201 1). NSF also suggested a host of potential partner agencies, both 
domestic and foreign, that could provide additional funding. Finally, to support foreign investigators 
from developing countries who were working with U.S. PIRE grantees, NSF partnered with USAID 
to establish the Partnerships for Enhanced Engagement in Research (PEER) Science program. 
Together, PIRE and PEER are jointly funding collaborations between U.S. investigators and their 
counterparts in developing countries where science and engineering capacity is emerging (USAID, 
2013). 


^ In comparison to OISE’s other investments, PIRE received nearly twice as much as OISE’s next largest 
investment priority, namely a combined $10.6 million allocated to three international programs for students 
and early career researchers: International Research Experiences for Students (IRES); East Asia and Pacific 
Summer Institutions (EAPSI); and International Research Fellowship Program (IRFP). 
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2. Methodology 


For the evaluation of PIRE, we used a combination of existing and primary data sources and 
descriptive, comparative, and quasi-experimental methods to explore four key areas: 

1. Research outcomes, including the quantity and quality (i.e., citation impact) of projects’ 
publications and the percentage of foreign coauthors of these publications; 

2. Participant experiences in the (PIRE or comparison) project, in particular, the nature of project- 
related activities conducted outside the U.S., collaborations between U.S. and foreign participants 
both during and subsequent to project participation, and perceived benefits and challenges of 
PIRE; 

3. The role of U.S. institutional policies and practices surrounding international faculty and student 
engagement and how these policies and practices affected — or were affected by — the PIRE 
project; and 

4. The potential contributions of PIRE to global societal challenges, in particular, challenges that 
may require international cooperation and coordination. 

A critical part of the evaluation methodology was the construction of a matched comparison group of 
NSF-funded projects that were not required to include an international collaboration and that were 
matched to PIRE projects along a set of key criteria such as award amount and duration. 

This chapter provides a brief overview of the data sources and analytic methods; the construction of 
the comparison group; survey sampling and response rates; and characteristics of the respondents. 
Volume 2 of this report provides additional technical detail on the data sources and methods. 

2.1 Data Sources 

Exhibit 2.1 summarizes the data sources used to address each of the evaluation’s research questions. 
In this section, we briefly describe each of these data sources. 

2.1.1 NSF Administrative Data 

NSF maintains administrative data at the aggregate level in relational databases for active and expired 
awards funded under current and former NSF programs and at the individual award level, including 
proposals and annual and final reports. We used the following NSF administrative data: 

• Award databases: To construct the comparison group of non-PIRE projects, we queried NSF 
award databases to identify candidate comparison projects that met specific criteria such as the 
funding directorate, award duration, type and grant amount. 

• Program solicitations: Having identified candidates, we used program solicitations as well as 
candidate and PIRE projects’ proposals and annual and final reports to perform additional 
screening against other key matching criteria. 

• Annual and final reports: We used annual and final reports to collect characteristics of individual 
PIRE and comparison projects and participants including Pl-reported publications resulting from 
the project (to inform bibliometric analyses) and the names of project participants and their 
institutions (to construct survey sampling frames).^ 


^ Participants listed by a PI in an annual and final report did not necessarily receive funding directly from the 
PIRE (or comparison) award to the PI(s). For example, graduate students might have had graduate 
fellowships to support their participation in the project; some undergraduate students had participated as 
part of a separate, NSF-funded Research Experience for Undergraduates (REU) grant to one or more Pis. 
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Exhibit 2.1 : Research Questions for the Evaiuation of NSF’s PiRE Program 


Research Questions 

Project or 
Participant 
Levei 


Survey Data 


Anaiyses 

NSF Administrative 
Data 

Pi and co-Pi 

Postdoc 

Graduate Student 

Undergraduate 

Foreign investigator 

Institutionai 

Representative 

Bibiiometric Data 

RQ 1 . What is the quantity and quaiity of the pubiications that PIRE projects 
have produced? 

Project 

X 

X 






X 

Descriptive 

RQ 2. How does the quantity and quaiity of pubiications produced by PIRE 
projects— that are required, by definition, to inciude an internationai 
coiiaboration— compare to the quantity and quaiity of pubiications 
produced by simiiar NSF-funded projects that do not require an 
internationai coiiaboration? 

Project 

X 







X 

Descriptive 

Comparative 

RQ 3. (a) What are the research outcomes for PiRE Pis, postdoctoral 

researchers, and graduate students? (b) What are the program 
experiences and educational or career outcomes of PIRE Pis, 
postdoctoral researchers, and graduate and undergraduate student 
participants? 

Participant 

X 

X 

X 

X 

X 



X 

Descriptive 

Comparative 

RQ 4. (a) How do the research outcomes for PIRE Pis, postdoctoral 

researchers, and graduate students compare to those of similar 
participants in similar NSF-funded projects that do not require an 
international collaboration? (b) How do the program experiences and 
educational or career outcomes of PIRE participants compare to similar 
participants in this comparison group? 

Participant 

X 

X 

X 

X 




X 

Descriptive 

Comparative 

Quasi-experimental 

RQ 5. How do international affairs representatives at PIRE lead Pis' 

institutions perceive the effects of PIRE on their institutions' policies 
and practices for supporting international research and educational 
collaborations? 

Project 







X 


Descriptive 

RQ 6. What are the program experiences of foreign senior investigative 

partners in PIRE and how do they perceive the effects of PIRE on their 
research and educational practices and those of their institutions? 

Participant 

X 





X 



Descriptive 

RQ 7. How have PIRE projects contributed to research that may inform global 
societal challenges? 

Project 


X 




X 



Descriptive 
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2.1.2 Survey Data 

The study included survey data collection from six respondent groups including current or former 
participants and administrators from each PIRE project’s sponsoring institution who did not 
participate directly in the projects: 

1. Principal and co-Principal Investigators (Pis) in PIRE and comparison projects; 

2. Postdoctoral participants (postdocs) in PIRE and comparison projects; 

3. Graduate student participants in PIRE and comparison projects; 

4. Undergraduate student participants in PIRE projects; 

5. Eoreign investigators (Els, i.e., non-U.S. personnel at a level equivalent to a U.S. PI) in PIRE 
projects; and 

6. Administrative representatives from lead PIRE institutions. 

Surveys of participants included questions about international travel, types of research and 
educational activities, and the nature of collaborations with foreign personnel both during and after 
their participation in the project. We asked PIRE participants about their perception of the benefits 
and challenges of PIRE, and PIRE Pis and Els about any potential contributions of the PIRE project 
to global societal challenges. Erom administrative representatives, we sought information about 
institutional policies and practices related to international research and education by faculty and 
students, how these policies might have facilitated or hindered PIRE activities, and whether the PIRE 
award had led to any changes in policy or practice. 

2.1.3 Bibliometric Data 

Eor each of the 59 PIRE projects and 55 matched comparison projects, bibliographic information for 
journal articles (i.e., authors, year of publication, article title, journal name and volume) was 
identified from lists of publications that Pis submitted to NSE in annual reports. These records were 
then submitted for comparison by Thomson Reuters Scientific staff to records in the Web of Science 
databases."^ For each Pl-reported record that matched a record in the Web of Science, we asked 
Thomson Reuters to provide the following data (see Definitions, below): 

• Field normalized citation impact (NCI); 

• Journal normalized citation impact (NCI); 

• Percentage of foreign institutions per article; 

• Percentage of foreign authors per article (for articles published 2009 or later); 

• Percentage of foreign institutions per article citing PIRE/comparison publications; and 

• Percentage of foreign institutions per article cited by PIRE/comparison publication(s). 

To investigate research outcomes for individual participants in PIRE and comparison projects, we 
collected the names of PIRE Pis, postdocs and/or graduate students, and when possible, email 
addresses and/or a bibliographic record for a selected “seed publication” when no email address was 


Although investigators disseminate their research in publications other than peer-reviewed journal articles, 
we restricted certain analyses of research outcomes to the data available on articles indexed in articles in 
Thomson Reuters’ Web of Science. We discuss the limitations of this restriction in Section 2.6. 
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available in NSF data. Thomson Reuters attempted to find publications on which participants were 
authors by matching the participant’s name and/or email address and using seed publications when 
available to disambiguate individuals with identical or similar names. Searches covered publications 
published in 2000 or later. For each article matched to a participant, Thomson Reuters provided: 

• Bibliographic data for each article published in 2000 through 2014 on which the participant was a 
coauthor; 

• Field normalized citation impact (NCI) per year, for each year from the year of publication 
through 2013; 

• Journal normalized citation impact (NCI) as of 2013; 

• Percentage of foreign institutions per article; and 

• Percentage of foreign authors per article (for articles published 2009 or later). 

Analyses of research outcomes for participants were restricted to articles co-authored by a PIRE or 
comparison project participant (PI, postdoctoral researcher, or graduate student). Articles were 
classified into two time periods: 

• Post-onset: those published in or subsequent to a participant’s first year of participation in the 
PIRE or comparison project (note that these publications could include articles or reviews that did 
not result directly from the PIRE or comparison project); and 

• Pre-onset: those published in the years prior to a participant’s first year of participation in the 
PIRE or comparison project. 

Eor participants who completed a study survey, we used the year in which the respondent reported 
first working on the project as the “onset” year. For survey non-respondents, we used the award start 
year. By definition, pre -onset publications did not result from participation in the project; post-onset 
publications could include both articles that resulted directly from the PIRE project and those that did 
not. Eurthermore, some participants had no pre -onset publications, some had no post-onset 
publications, and some had neither. (See Volume 2 for results of participant-level matching.) 

2.2 Constructing the Counterfactual 

Eor each of the 59 PIRE projects awarded in the 2005, 2007, 2010 and 2012 cohorts, we attempted to 
identify a similar NSF project that did not require an international collaboration, but wbicb was 
similar to its corresponding PIRE project along other key criteria, following a series of iterative steps. 
Eirst, we generated candidate awards from the NSE directorates that most closely matched the PIRE 
award’s research area; had an award amount (including any supplementary funding) within 20 percent 
of the PIRE award’s amount; had award start and end dates within 12 months of those of the PIRE 
award; were funded as continuing grants; and had a duration (including any extensions) within 12 
months of the PIRE project’s duration (or expected duration for active projects). Next, using award 
abstracts, proposals, and annual and final reports, we screened each candidate project further to 
ensure that: 

• The project was funded by an NSE program whose primary purpose was research (in contrast to 
programs focusing on, e.g., training, workforce diversity or equipment or infrastructure). 

• The project included at least two different participating institutions (U.S. or foreign). 

• The project included at least one graduate student. 
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• The PI was not a current or former PIRE PI. 

• The applicable program solicitation(s) either encouraged applicants to propose an international 
collahoration; mentioned international collahoration as an option; or did not mention international 
collahoration at all. ^ 

• In addition, we carefully scrutinized the candidate’s award’s abstract, annual reports, and the CVs 
of Pis, co-PIs and senior personnel to identify candidate projects where the research area(s) was 
similar to the PIRE award. To make this determination, we looked for evidence from award 
abstracts, reports and CVs of senior personnel that the comparison project would require types of 
knowledge or expertise similar to that required by the PIRE project. 

If multiple candidates meeting these criteria were identified, the comparison project that best matched 
the research focus of the corresponding PIRE award was selected. Einally, if no comparison award 
was eligible, we attempted to generate new candidates (i.e., the first step) by including additional 
directorates (if likely to yield viable new candidates) in the search query and: 

• Allowing the start and end dates of candidate projects to be within 24 months of the PIRE award 
(n=10 comparison projects); 

• Allowing the duration of candidate projects to be within 24 months of the total PIRE duration 
(n=7 comparison projects); 

• Allowing the candidate award amount to differ by up to 25 percent of the PIRE amount (n=5 
comparison projects); and 

• Including standard grants as an eligible award type (n=10 comparison projects). 

We identified a matching project for 55 of the 59 PIRE projects from a population of approximately 
1,500 projects that met step one criteria. Volume 2 summarizes the resulting characteristics of the 
matched pairs of projects. Eor four of the PIRE awards, no matched comparison project was found. 

To protect the identity of the selected comparison projects and personnel, we withheld the award 
numbers, project titles, institutions, and other identifying characteristics from NSE.® 

2.3 Sampling, Locating Respondents, and Response Rates 

To construct the sampling frames for the online surveys, we obtained names and institutional 
affiliations of PIRE and comparison project participants from Pl-submitted annual reports. Eor PIRE 
and comparison projects, we selected a census of Pis postdoctoral researchers, graduate students and 
foreign investigators identified from annual project reports. We selected a sample of undergraduates 
proportional to the number of undergraduates participating in each PIRE project, and a convenience 
sample of institutional administrators from lead Pis’ institutions (i.e., from the PIRE award’s 
sponsoring institution). 

We refer to all participants who were primarily based at a U.S. institution at the time of participation 
as “U.S. -based” participants. This designation is unrelated to participants’ actual citizenship. Thus, a 


^ An NSF program that disallowed international collaborations would have been ineligible but no such 
program was encountered. 

® The purpose of the comparison group was to evaluate PIRE against a counterfactual, not to provide direct 
comparisons of the outcomes of any individual comparison project to its matched PIRE project. 
Withholding the comparison project identities helped ensure participation in the evaluation by comparison 
project personnel. 
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U.S. postdoctoral participant in PIRE or a comparison project could have been a U.S. citizen or not. 
Likewise, a “foreign investigator” was defined based on his or her primary institutional affiliation: if 
this institution was outside the U.S. the individual was a foreign investigator, regardless of actual 
citizenship. (Note that project participants were not necessarily receiving funding directly from the 
project in which they participated.) 

2.3.1 Respondent Group 

Individual participants were classified based on their “status” (as faculty, postdoc, or student) when 
they first participated in the project. We based this classification on their reported status in the earliest 
annual report in which they were listed: for example, if an individual participated as both a 
postdoctoral researcher and a graduate student, the individual was classified as a graduate student 
participant. During survey fielding, we reclassified individuals as needed based on screening 
questions in the survey. For example, two individuals identified as graduate student PIRE participants 
in our sample indicated that they had been undergraduates when they first participated in the project. 
We reclassified these individuals and invited each to complete the PIRE undergraduate survey. We 
excluded respondents whom we determined were ineligible for the survey (e.g., individuals who 
indicated that they were not official co-PIs or foreign graduate students). 

2.3.2 Locating Respondents 

We supplemented existing contact information from annual reports with information publicly 
available via internet searches (e.g., departmental web pages) and by asking Pis to provide the most 
recent email address (or other contact information) for both current and former participants. To 
identify institutional administrators, we asked lead Pis to nominate up to two appropriate 
administrative officials who would be familiar with the PIRE award (e.g., a director of international 
affairs, or vice provost for research); searched institutional web sites for names of officials in 
administrative offices; and allowed administrators invited to complete a survey to indicate an 
alternative official who might be better suited to complete the survey. 

2.3.3 Response Rates 

Exhibit 2.2 summarizes the response rates for each survey. We sent an emailed survey invitation and 
link to each targeted sample member and six reminder emails. For Pis and FIs, we also mailed letters 
to institutional addresses to encourage survey participation. During survey fielding, we determined 
that some sample members were ineligible for the survey; we excluded them from response rate 
calculation and analyses. (Misclassified respondents were invited to participate in the survey 
corresponding to their correct respondent group.) 


Exhibit 2.2: Sampie Size and Response Rates for Study Surveys 


Respondent Group 

PIRE 

Comparison 

Sample Size 

Response Rate 

Sample Size 

Response Rate 

Lead Pis 

59 

71% 

55 

62% 

All Pis 

293 

58% 

205 

56% 

Postdocs 

211 

55% 

235 

50% 

Graduate Students 

531 

54% 

705 

47% 

Undergraduates 

735 

40% 

NA 

NA 

Foreign Investigators 

251 

47% 

NA 

NA 

Institutionai Administrators 

59 

78% 

NA 

NA 


Sources: Annual reports submitted by Pis to NSF; study surveys. 

Notes: 

Sample size is equal to the number of eligible respondents. The response rate indicates the percentage of this sample that 
completed a survey. Because it is impossible to verify that the intended recipient has received an emailed survey invitation, 
response rates include sample members who may not have received the survey invitation. 
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Lead Pis and institutional administrators for PIRE projects had the highest response rates (71 and 78 
percent) and undergraduates had the lowest (we had particular difficulty finding a valid email address 
for undergraduates). In general, response rates for comparison respondent groups were lower than for 
the corresponding PIRE groups. 

2.4 Respondent Characteristics^ 

Roughly one -quarter to one-third of PIRE and comparison Pis and post-docs were female. Among 
graduate and undergraduate students, 40 to 45 percent were female. Only 19 percent of Els were 
female. Less than one -fifth were members of racial or ethnic groups traditionally under-represented in 
science and engineering. 

The majority of participants reported that their primary field of research (or the field in which they 
were pursuing an undergraduate or graduate degree) was in the sciences, mathematics or engineering. 
Notably, 24 percent of PIRE Pis reported that their primary research field was in engineering and 14 
percent that it was in the social sciences; in both cases, the percentage was twice that of comparison 
Pis. A greater proportion of comparison Pis were in the natural sciences or mathematics. 

On average, postdoctoral researchers reported participating for just under two years in the project 
(PIRE or comparison). Undergraduate and foreign senior investigators each reported an average of 
nine months of participation. Graduate students took part in a PIRE project for an average of 31 
months (just over 2.5 years), whereas graduate students in comparison projects reported an average 
duration of more than 35 months (just under 3 years). 


^ We did not conduct statistical tests of significance for apparent differences between PIRE and comparison 
group characteristics reported here. 
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Exhibit 2.3: Characteristics of Participants 



Pis 

Postdocs 

Graduate 

Students 

Under- 

graduates 

FIs 

PIRE 

COMP 

PIRE 

COMP 

PIRE 

COMP 

PIRE 

PIRE 

Demographic characteristics 




Female^ 

23% 

25% 

36% 

27% 

41% 

40% 

45% 

19% 

Member of minority group 
traditionaiiy under-represented 
(URM) in S&E2 

10% 

16% 

17% 

15% 

17% 

16% 

20% 

NA 

Primary field of research/field of study^ 



Natural Sciences or Mathematics 

61% 

81% 

82% 

83% 

60% 

70% 

65% 

77% 

Engineering 

24% 

12% 

12% 

15% 

31% 

22% 

24% 

18% 

Social Sciences 

14% 

7% 

6% 

<1% 

9% 

8% 

4% 

5% 

Health 

<1% 

0% 

0% 

<1% 

0% 

<1% 

2% 

0% 

Humanities, Arts, 
Business/Management or Other 

NA 

NA 

NA 

NA 

NA 

NA 

4% 

NA 

Average duration, in months, of 
participation in the project 
(standard deviation)'* 

12 

(11.0) 

12 

(10.2) 

23 

(13.2) 

22 

(13.1) 

31 

(20.4) 

35 

(18.8) 

9 

(11.9) 

9 

(11.5) 


Sources: 

Percent female, Percent URM: PI survey. Items H3-H5; Postdoc survey. Items F3-F5, Graduate survey, Items F3-F5; 

Undergraduate Survey, Items G3-G5; FI Survey, Item G4 (percent female). 

Primary research discipline/field of study: PI survey, Item E3; Postdoc survey. Item D2; FI Survey, Item AI2. Primary field 

of graduate degree: Graduate survey. Item A3; Primary major field for bachelor ’s degree: Undergraduate .survey. Item A 7. 

Duration: PI survey. Items A3-A7; Postdoc survey: Items Cl, Cla, CIb; Graduate student survey. Items Cl, Cla, Clb; 

Undergraduate survey: Items B7, B7a, B7b; FI survey: Items A3-A7. 

Notes: 

* Percent female: Pis: PIRE N=163, missing=7; Comparison N=1 12, missing=2. Postdocs: PIRE N=1 13, missing=9; 
Comparison N=1 13, missing=8 . Graduate students: PIRE N=278, missing =7; Comparison N=323, missing=8. 
Undergraduates: N=286, missing=4; Els: N=I04, missing=I8. 

^ Percent URM: Survey respondents who reported race/ethnicity as Black/ African- American, Hispanic, American 
Indian/ Alaska Native, Native Hawaiian/other Pacific Islander were classified as traditionally under-represented 
minorities (URM). Pis: PIRE N=I62, missing=8. Comparison N=I 12, missing=2; Postdocs PIRE N=1 1 1, missing=I I, 
Comparison N=I 12, missing=9; Graduate students: PIRE N=278, missing=7. Comparison N=323, missing=8. 
Undergraduates: N=283, missing=7. 

^ Primary research discipline: Pis: PIRE N=170, Comparison N=I 14; Postdocs: PIRE N=1 14, missing=8. Comparison 
N=1 13, missing=8; FIs: N=I2I, missing=I. Primary field of study for graduate degree: Graduate students: PIRE 
N=285, Comparison N=331; Primary major field for bachelor’s degree: Undergraduates: N=246. A list of disciplines 
classified as natural sciences, mathematics, engineering, social sciences and health is in Volume 3 of the report. Only 
undergraduates had the option to indicate that their major field for their bachelor’s degree was in Humanities or Arts or 
in Business or Management; Pis, FIs, postdocs and graduate students could select from natural sciences or 
mathematics; engineering; social sciences; or health. 

Duration, Pis: PIRE N=168, missing=2. Comparison N=1 14; Postdocs: PIRE N=I07, missing=I5, Comparison N=I08, 
missing=I3; Graduate students: PIRE N=275, missing=10. Comparison N=327, missing=4. Undergraduates: N=207, 
missing=2; FIs: N=I21, missing=l. For Pis and FIs, the duration in months was calculated by multiplying the percent 
time a PI reported spending on the project by the elapsed time between the reported first date of participation and the 
date on which they last contributed to the project. In active projects, we set the date of last contribution to the project 
for participants who reported that they were still contributing to the project to the survey reference date. 
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2.5 Analysis Methods 

2.5.1 Analyses of Research Outcomes (Chapter 3) 

Analyses of publication outcomes (quality, quantity, and international collaborations) were conducted 
at tbe project level (i.e., for projects as a whole including all journal articles reported by tbe PI as 
resulting from tbe project), and at tbe participant level (i.e., analyzed separately for each of three 
types of project participants: Pis, post-docs, and graduate students). At the project and participant 
levels, analyses included: (a) a descriptive analysis of outcomes for PIRE projects (or PIRE 
participants), and (b) comparative descriptive analyses of the differences between PIRE and 
comparison projects (or between PIRE and comparison group participants). At the participant level, 
analyses also include (c) an impact analysis comparing outcomes for PIRE and comparison 
participants. 

Descriptive analyses 

The descriptive analyses show the distributions of key publication outcomes across PIRE projects 
(Research Question 1) or PIRE participants (Research Question 3). The descriptive statistics reported 
include the mean, standard deviation, minimum, maximum and quartiles (25* percentile, median, 75* 
percentile) for each outcome. Where informative, we also include graphical displays that provide a 
more fine-grained illustration of these distributions than indicated by quartiles. At the project level, 
these analyses included all PIRE projects where at least one publication record was identified by 
Thomson Reuters’ Web of Science (51 PIRE projects). At the participant level, these analyses 
included all Pis, postdocs and graduate students who had at least one post-onset publication identified 
in the Web of Science (265 Pis, 135 postdocs and 181 graduate students).* (Note that articles 
published post-onset may have resulted from activities other than participation in the PIRE project.) 

Descriptive comparative analyses 

We also include descriptive comparative analyses to show differences in research outcomes for PIRE 
and comparison projects (Research Question 2) and participants (Research Question 4). A comparison 
project was identified for 55 of the 59 PIRE projects. However, at the project level, these descriptive 
comparative analyses included only the subset of 45 matched pairs of projects where each project 
within each pair had at least one journal article identified in the Web of Science. Ordinary least- 
squares (OLS) regression analyses included the outcome as the dependent variable; independent 
variables included a group indicator (i.e., l=a PIRE project; 0=a comparison project) and dummy 
variables to account for the matched pairs of projects. 

At the participant level, we conducted a descriptive comparative analysis of the percentage of foreign 
affiliations among co-authors of participants’ post -onset publications. These analyses included only 
participants from within matched project pairs who had at least one post-onset and at least one pre- 
onset publication indexed in the Web of Science. The analysis used ordinary least squares regression 
models that included a group indicator (PIRE=1, COMPARISON=0) and dummy variables 
representing the matched pairs of projects; these participant-level analyses also controlled for the pre- 
onset measure of the outcome (e.g., percentage of foreign institutions per publication per participant). 

Impact analyses of participant-level outcomes 

To estimate the impact of PIRE on participant-level research outcomes (Research Question 4), we 
used a quasi-experimental, comparative interrupted time series (CITS) design to compare outcomes of 


Post-onset publications are those published in the year of or after the start of participation in the project; see 
Definitions. See Volume 2 for further details on matching of project publications and project participant 
names to the Web of Science records. 
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PIRE participants to those of matched participants from the group of comparison projects (see 
Volume 2 for technical details). We applied this method to test the impact of PIRE on two research 
outcomes — the number of publications and the field normalized eitation impaet (field NCI) for post- 
onset publications.^ We conducted separate analyses for Pis, postdoctoral researchers and graduate 
students. Eor each group, we matched PIRE participants to corresponding participants from 
comparison group projects on their first year of projeet partieipation and attempted to limit the 
analysis sample to participants who were statistically equivalent on both number of publications and 
average field NCI per year for publieations prior to the onset of projeet partieipation. The CITS 
analysis for eaeh group ineluded four years of pre-onset data and multiple years of post-onset data. 

A CITS analysis is similar to a differenee-in-difference model with data at multiple “pre -treatment” 
and “post-treatment” time points. The analysis calculates the difference between (a) the pre -post 
differenee in the outcome measure for PIRE partieipants and (b) the pre -post differenee in the 
outcome measure for comparison project participants. 

Data from participants in comparison projects — who were producing publications in the same years 
as their PIRE eounterparts — aets as a eontrol for faetors external to PIRE that could have affeeted 
publieation productivity or citation impacts (e.g., macroeconomic trends affecting research and 
development efforts in general, independent of any effects of PIRE on research outcomes). Data from 
partieipants prior to the onset of their partieipation in a projeet (PIRE or eomparison), aets as a 
eontrol for faetors related to time -invariant differenees between partieipants that eould affeet the 
research outcomes independent of the effects of PIRE participation on these same outcomes (e.g., 
differences between participants’ research disciplines are well known to affeet publieation trends and 
it is reasonable to assume that the effeet of research discipline is stable aeross time). By eontrolling 
for two potential sources of observed differences in research outcomes that are independent of PIRE, 
the CITS method reduees the number of plausible alternative explanations for observed differenees, 
and increases confidence that differences likely result from the effect of PIRE. 

We note three important caveats about the analyses of research outcomes: 

• The size of analysis samples differs for the deseriptive, eomparative and impaet analyses. That is, 
at the project level, as a result of taking into account the matched project pairs and restricting the 
sample to projeets with at least one publieation indexed in the Web of Scienee, the sample for 
eomparative analyses ineludes just the 45 matched pairs of PIRE and eomparison projeets. At the 
participant level, comparative samples include only participants within matched pairs who had at 
least one pre -onset and one post-onset publieation indexed in the Web of Seienee (Pis: PIRE 
N=204; eomparison N=168 in 54 matehed projeet pairs; postdoes: PIRE N=48; eomparison N=55 
in 34 matched project pairs; graduate students: PIRE N=36; comparison N=56 in 45 matched 
projeet pairs)." 


® Post-onset articles are those published after the participant had begun participating in the PIRE or 

comparison project. Pre-onset articles are those published prior to this onset of participation. (See Section 
2.1.3: Data Sources, Bibliometric Data.) 

*** We defined baseline equivalence for annual number of publications and field NCI per year as a 

standardized difference in pre-onset means between the two groups of less than .25 standard deviations. We 
established equivalence for all respondent groups on each measure with one exception: the PI analysis 
sample was not equivalent for number of pre-onset publications per year, but was equivalent on the pre- 
onset mean field NCI per year. 

" A pair of matched projects from the total of 55 was dropped if there were no respondents with pre- and 
post-onset publications in one or both of the members of a pair of projects. For example, if a PIRE project 
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• Unlike the quasi-experimental impact analyses, the descriptive comparative analyses do not allow 
any inferences about the underlying causes of observed differences in research outcomes: that is, 
from these analyses, we cannot conclude that the differences in research outcomes were caused 
by the PIRE program. 

• In addition, because the impact analyses are quasi-experimental rather than experimental, 
attributions of differences in research outcomes to the PIRE program must acknowledge the 
possibility of alternative explanations. We discuss this third caveat in more detail below (see 
Section 2.6: Limitations). 

2.5.2 Analyses of Participant Experiences (Chapter 4) 

We conducted both descriptive and descriptive comparative analyses of primary survey data. 
Descriptive analyses included both quantitative and qualitative methods. Eor continuous variables we 
report means, standard deviations and, where informative, other characteristics of variation around the 
mean such as the median, minimum, maximum and inter-quartile values. Eor categorical and 
dichotomous variables, we report frequencies in terms of number and percentages. Missing data were 
excluded from these calculations. Where possible, we re-coded responses to “other, specify” response 
options into an existing response provided; if the response did not fit an existing response option we 
counted it as “other.” For open-ended survey items, we used NVivo software to explore similarities 
and differences across respondents and identify and summarize key themes that emerged from these 
data. We identified emergent themes inductively using an iterative coding approach in which we 
generated potential themes from an initial review of a subset of responses and then applied these 
themes to the entire set of responses, iterating between generating themes and coding responses 
against those themes. 

To compare the experiences of PIRE and comparison Pis, postdocs, and graduate student participants, 
we also conducted descriptive comparative analyses for selected outcomes. These analyses used 
ordinary least squares (OLS) regression analyses with the outcome as the dependent variable; 
independent variables included a group indicator (i.e., l=a PIRE project; 0=a comparison project) and 
dummy variables to account for the matched pairs of projects (N=55). Because we matched projects 
on criteria such as award amount, duration, award start and end years, and research area(s), 
controlling for these pairs effectively controls for these matching criteria. 

Although these comparative analyses provide a test of the statistical significance of differences 
between PIRE and comparison participants’ outcomes, these analyses do not support inferences about 
the underlying cause of these differences. Differences may or may not have resulted from 
participation in the PIRE program or the programs represented by the comparison group. 

2.5.3 Analyses of Institutional Policies and Practices for International Research and 
Education (Chapter 5) 

Analyses of the PIRE institutional administrator survey data (as well as some data collected from Pis 
about their institutions) include both quantitative and qualitative descriptive analyses like those 
described above for participant experiences. Where useful, we include direct quotations to illustrate 
key themes. (We have deliberately censored the names of individuals, institutions, and geographic 
locations to protect the anonymity of respondents and their institutions.) 


had one postdoc who had pre and post onset publications but the matched comparison project had a postdoc 
with no pre-onset publications, the pair of projects was dropped from the analysis for postdocs. 
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2.5.4 Analyses of Societal Impacts (Chapter 6) 

To summarize data bearing on the potential contributions of PIRE to global societal challenges, we 
conducted primarily qualitative data analyses of open-ended survey responses to illuminate key 
themes as well as variation or notable divergence from these themes. We include direct quotations 
(with censoring to protect anonymity of respondents) where meaningful. 

2.6 Limitations 

Below, we describe three key limitations of the study and how these suggest caution in interpretation 
of the study findings. These key limitations are: 

1. In a non-experimental study like this one, there is a risk that omitted variables, rather than the 
PIRE program itself, may be responsible for any observed differences between PIRE and non- 
PIRE outcomes. 

2. The sole source of participant information - annual and final reports to NSE - often lacked 
contact information, and in some instances names of participants were also missing, particularly 
the names of participants from institutions other than the lead Pi’s. 

3. We obtained lower than expected survey response rates and did not have data needed for a non- 
response bias analysis or weighting of the survey data for non-response. 

4. Our methods for assessing research outcomes based on bibliometric data are limited by the 
following: 

a. We could only obtain bibliometric data for peer-reviewed journal articles; 

b. We used citation indices as one indicator of the “quality” of publications, despite the fact that 
citation indices are influenced by factors other than the underlying quality of the research; 
and 

c. The bibliometric data matching procedures we used likely reflect some omissions or 
inaccuracies in the reference data listed in Pis’ annual reports and the fact that more recent 
PIRE and comparison projects have had less time to accrue publications and citations. 

2.6.1 Non-Experimental Comparison Group 

The most rigorous estimates of the impact of PIRE on outcomes of interest would come from a study 
in which the PIRE and comparison group were determined using random assignment. Because award 
decisions for past cohorts of the program had already been made when the evaluation was planned, 
random assignment of applicants to treatment (receive a PIRE award) or control (do not receive a 
PIRE award) was infeasible. Even if the evaluation had been conducted prospectively (i.e., prior to 
award decisions), it is unlikely that random assignment would have been feasible given NSF’s 
proposal and award procedures.'^ We also determined that the most rigorous quasi-experimental 
designs were not possible (e.g., a regression discontinuity design, or propensity score matching of 
awarded to non-awarded PIRE proposals based on pre-award characteristics).*^ Instead, we 


Even in merit-based programs, it is possible to conduct a random assignment study if there are a greater 
number of equally qualified applicants than available funding or award openings. However, NSF’s proposal 
review includes consideration of both intellectual merit and broader impacts, and program officers exercise 
discretion in making awards to meet other foundation-wide objectives (e.g., such as increasing the 
representation of awards to under-represented minorities or institutions). 

Further discussion of the rationale for the study design is included in Volume 2. 
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constructed the strongest possible non-experimental comparison group using available data sources 
(i.e., other NSF awards), and matching a comparison project to each PIRE project as closely as 
possible across criteria thought to most strongly influence project outcomes (e.g., award amount, 
duration, research field(s), year of award onset). Nevertheless, as with any non-experimental 
comparison group, the possibility remains that we did not take into account some other set of factors 
that cause the outcomes of PIRE and the comparison group to differ. This risk (sometimes called 
“omitted variables bias”) means that the majority of our reported findings do not rule out alternative 
explanations for differences between PIRE and comparison group performance. As a result, we urge 
caution in interpreting differences between PIRE and the comparison group. It is possible, for 
example, that pre-existing differences in the Pis and other individuals in the PIRE and comparison 
group are responsible for differences in the project- and participant-level outcomes, and that these 
differences may have occurred even in the absence of the PIRE program. 

One exception to this caveat is our analysis of the research outcomes of PIRE participants relative to 
those of comparison group participants. This analysis used a robust quasi-experimental comparative 
interrupted time series design. These analyses not only take advantage of the project-level matching 
of comparison and PIRE projects, but also matched individual participants on pre -project 
characteristics associated with the outcomes. 

2.6.2 Incomplete Participant Information in Annual Reports 

The primary — and to our knowledge, only — sources of the names of individual participants in PIRE 
and comparison projects were annual and final reports that Pis submit to NSE. However, for many 
such participants, these reports lacked contact information necessary to invite the participant to 
complete a study survey. In addition, without an email address for a participant, it was difficult to 
match participant names to publications in the Web of Science on which they were potential co- 
authors. Especially for researchers with common names, disambiguating multiple authors with the 
same (or highly similar) name becomes difficult without a unique identifier such as an email address. 
We attempted to collect email addresses by conducting internet searches and by asking co-PIs to 
provide contact information for participants listed in the annual reports. Several also supplied the 
names (and when possible, an email address) of additional participants who had been omitted from 
these reports. The lack of participant contact information was especially serious for undergraduates: 
of 735 undergraduate participants in our survey sample, we were unable to locate an email address for 
196 (more than 25 percent). 


2.6.3 Survey Response Rates 

Despite providing multiple prompts to encourage survey participation, we obtained lower than 
expected response rates for postdoctoral researchers, graduate students, Els and undergraduates. In 
particular, just 40 percent of the undergraduate sample completed a study survey. Part of the reason 
for lower-than-expected response rates stemmed from the fact that NSE lacks a systematic mechanism 
for collecting program participants’ contact information other than annual project reports, and no 
mechanism for keeping track of program “alumni” whose participation in a program has ended. 
Although former graduate students or postdocs often remain in academia and can be found via their 
affiliation with a university, undergraduates and other former participants in NSE programs who go 
on to jobs in industry are difficult to locate. As a result, the survey responses we received from former 
participants likely over-represent those who remained in academia (e.g., undergraduates who enrolled 
in graduate school) and under-represent participants now working in industry. Moreover, because we 
could not identify data on the characteristics of sample members (e.g., demographic characteristics) 
we were unable to perform a non-response bias analysis. It is possible that survey respondents differ 
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systematically from non-respondents and we suggest caution in interpreting the survey findings from 
these groups. 

2.6.4 Methods For Assessing Research Outcomes 

A key objective of the evaluation was to compare the quality of research produced hy PIRE to the 
quality of research produced hy the comparison group. We used measures of citation impact to 
represent the quality of research in spite of the well-known limitations of citation impacts as a 
measure of quality (Bornmann et ah, 2008; Leydesdorff, 2008; Moed, 2005; Pendlehury, 2009). 
Nevertheless, given the sheer number of publications included in the evaluation, we judged that an 
alternative, such as expert review of publication quality, was not feasible. Instead, we have attempted 
to follow the principles of the “Leiden Manifesto” (Hicks et al., 2015) by including additional 
measures that align with the PIRE program’s key goals to stimulate research collaboration between 
U.S.- and non-U.S. -based researchers (e.g., the percentage of foreign co-authors per paper); by using 
normalized citation indices to account for variation across research disciplines in publication and 
citation trends; and by clearly describing the analytic procedures and their limitations. 

Investigators disseminate their research in a variety of forms and venues, including book chapters, 
conference papers, open-access journals and repositories (e.g., arXiv), and peer-reviewed journals. 
However, because citation data were not readily available for other types of publications, we limited 
our analyses of research outcomes in this study to articles in peer-reviewed journals that are indexed 
in Thomson Reuters’ Web of Science. Restricting the data to peer-reviewed journal articles may be 
particularly problematic for disciplines such as computer science or mathematics, in which 
researchers rely less heavily on journal articles than on conference proceedings and online 
repositories. We note, however, that among the 59 PIRE projects and 55 comparison projects, only 6 
projects in the evaluation (3 PIRE, 3 comparison) were CISE-related, and 2 (PIRE) were 
mathematics -related. Nevertheless, some projects never intended to produce peer-reviewed journal 
articles, but rather, focused on contributing other resources such as data repositories, software, and 
similar products for dissemination; for these projects, our focus on journal articles excludes their 
primary research products. 

We also excluded articles in journals not indexed in the Web of Science for which citation data may 
have been available elsewhere (e.g., from Scopus or Google Scholar). Thomson Reuters was unable 
to match a small proportion of journal articles listed in Pis’ annual reports to existing records in the 
Web of Science despite indexing the journal in which the article was published. (See Volume 2 for 
more detail regarding the bibliometric data matching.) In particular, among the more recently 
awarded PIRE and comparison group projects (i.e., those with award onset in 2012 or later), annual 
reports were more likely to have fewer publications (of any kind) listed, and those identified in the 
Web of Science had had less time to accrue citations. 

Because of these limitations, we encourage caution when interpreting the findings reported for 
research outcomes based on the available bibliometric data. However, we note that we found no 
statistically significant differences between the proportion of PIRE and comparison projects’ 
publications that were matched to the Web of Science. 
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3. Research Outcomes 


PIRE seeks to enable research at the leading edge of science and recognizes the value of international 
partnerships in addressing critical science and engineering questions. Thus, a primary focus of the 
evaluation was to explore the research outcomes of PIRE projects and to understand how these 
outcomes compared to research outcomes of non-PIRE NSF projects. In this chapter we describe the 
quantity and quality of research produced by PIRE projects (and PIRE participants), as well as 
measures of international engagement in this research. We also examine whether and how these 
research outcomes differ from those of a matched comparison group of projects and for matched 
participants within the matched pairs of projects. Specifically, the findings address the following 
research questions: 

• RQ 1. What is the quantity and quality of the publications that PIRE projects have produced? 

• RQ 2. How does the quantity and quality of publications produced by PIRE projects — that are 
required, by definition, to include an international collaboration — compare to the quantity and 
quality of publications produced by similar NSF-funded projects that do not require an 
international collaboration? 

• RQ 3a. What are the research outcomes for PIRE participants (Pis, postdoctoral researchers and 
graduate students)? 

• RQ 4a. How do the research outcomes for PIRE Pis, postdoctoral researchers, and graduate 
students compare to those of similar participant groups in similar NSF-funded projects that do not 
require an international collaboration? 

At the project level, we report both descriptive analyses (Research Question 1) and comparative 
analyses (Research Question 2) that take into account the one-to-one matching of PIRE and 
comparison projects. At the participant level, we report descriptive findings for research produced by 
PIRE Pis, postdocs and graduate students from the onset of their participation in the project forward 
(i.e., post-onset publications); we present a descriptive comparative analysis of the contributions of 
foreign co-authors to PIRE and comparison project participants’ post-onset publications, taking into 
account matched pairs of projects; and finally, we present impact analyses of the effects of PIRE on 
Pis,’ postdocs,’ and graduate students’ post-onset publications. These impact analyses use data from 
participants’ pre-onset and post-onset periods to estimate the difference in the pre-post differences in 
research productivity and quality between these three groups of PIRE and comparison participants. 

Specific research-related measures we report include the: 

• Total and average annual number of peer-reviewed journal articles published; 

• Citation impact measures of these articles (field NCI and journal NCI); 

• Relative contribution of authors based at foreign institutions to these articles; 

• Extent of foreign co-authorship in the research that is cited by PIRE and comparison projects’ 
publications; and 

• Extent of foreign co-authorship of research that cites PIRE and comparison projects’ publications. 

Findings in this chapter are drawn from annual reports submitted by projects to NSF and the 
Thomson Reuters’ Web of Science database. We describe the data sources and the analysis methods 
used more fully in Chapter 2 (with supplementary technical details in Volume 2 of the report). 
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Key findings include: 

• Across PIRE projects, there is variation in research productivity. Six projects reported no 
publications and two others had no publications indexed in the Web of Science. Across the 51 
PIRE projects with publications recorded in the Web of Science, the average number of 
publications was 24. 

• Across PIRE journal articles the mean field NCI was 1.8 and the journal NCI was 1.3, reflecting a 
higher than average impact of these articles relative to the impact of an average journal article 
(the average for each NCI measure=l). 

• There were no statistically significant differences between PIRE and the comparison group in the 
quantity or quality of research produced. However, PIRE publications had a significantly higher 
proportion of foreign contributions per journal article measured both as the percentage of foreign 
institutions per article and the percentage of foreign authors per article. 

• At the participant level, there were no significant differences in annual number of post-onset 
publications between PIRE and comparison Pis. However, among postdocs, PIRE participants 
had, on average, a larger number of post-onset annual publications and these publications had a 
larger impact. PIRE graduate students also had a higher average number of annual post-award 
publications than their comparison group peers. 

• PIRE Pis’ post -onset publications had, on average, a statistically significant higher percentage of 
non-U.S. institutional affiliations among authors than comparison Pis’ post-onset publications. 

3.1 PIRE Projects’ Publications 

Findings used references submitted by Pis in annual and final reports. As described in Chapter 2, we 
excluded publications other than journal articles from these analyses. 

3.1 .1 Number and Quality of Publications 

PIRE projects varied considerably in terms of research productivity, as measured by their output of 
journal articles.'"^ PIRE-supported research resulted in a project average of 24 articles but the mean is 
skewed by the relatively small number of projects with much higher numbers of articles (the median 
is 22 articles). 

The average impact of research published by PIRE was larger than average, measured by either field 
or journal NCI. Exhibit 3.1 indicates that the distribution of the citation impacts is skewed; for 
example, the median field NCI across PIRE journal articles is 1.09, but the 95* percentile is 5.78. 
Thus, although half of these publications have an NCI higher than 1.09, 5 percent have a field NCI 
that is more than five times that of the median. 


For some PIRE projects, particularly those involving research related to computer and information 
sciences, using journal articles to examining research output underestimates their contributions, since these 
projects did not plan to publish research reports in journals, but rather were designed to produce resources 
such as data repositories and software accessible to a broad community of researchers. Moreover, not all 
PIRE projects intended to produce journal articles, but rather, set out to establish resources available to the 
broader research community (e.g., data repositories). 
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Exhibit 3.1 : Key Bibiiometric indicators for Articies Produced by PiRE Projects (N = 1,399 
articies) 


Bibiiometric 

Indicator 

Nof 

Articles 

with 

Data® 

Minimum 

Qlb 

Median*’ 

Q3*’ 

95* 

Percentile*’ 

Maximum 

Mean 

SD<= 

Number of articles 
per project<i 

1,399 

0 

2 

22 

42 

70 

88 

23.7 

23.5 

Field NCI 

1,339 

0.00 

0.35 

1.09 

2.19 

5.78 

45.05 

1.82 

3.04 

Journal NCI 

1,368 

0.00 

0.27 

0.80 

1.74 

4.11 

40.67 

1.30 

1.89 

Percentage of 
authors at non- 
U.S. institutions 

1,153 

0.0% 

0.0% 

28.6% 

60.0% 

100.0% 

100.0% 

32.7% 

32.5% 

Percentage of 

non-U.S. 

institutions 

1,398 

0.0% 

0.0% 

43.7% 

61.2% 

100.0% 

100.0% 

36.7% 

31.5% 


Sources: Annual reports submitted by Pis to NSF; Thomson Reuters ’ Web of Science 


Notes: 

“ Of the 1,399 articles identified in the Weh of Science, field NCI data were available for 96 percent, journal NCI for 98 
percent, percentage of foreign authors for 82 percent (unique author affiliations are only available in the Web of 
Science for articles published in 2009 or later), and percentage of foreign institutions for nearly 100 percent. 

Ql, the first quartile, indicates the value of the 25’’’ percentile for the given indicator (75 percent of the articles have 
values greater than the Ql value). The median indicates the 50* percentile (half of the articles have values greater than 
the median). Q3, the third quartile, indicates the value of the 75* percentile (25 percent of the articles have values 
greater than the Q3 value shown). At the 95* percentile, 5 percent of articles have values greater than shown. Taken 
together, the quartiles and 95* percentile illustrate the degree to which the distribution of an indicator is skewed (or 
not). 

SD=standard deviation 

Some PIRE projects never intended to produce journal articles, but instead produced data repositories, software and 
similar resources accessible to researchers but not reflected in bibiiometric counts of published articles. 

Across PIRE projects, the distribution of the number of journal articles is positively skewed, such that 
a small minority of projects produced a large number of publications, with most producing a small 
total number of publications (Exhibit 3.2 shows a more fine-grained illustration of this distribution 
than Exhibit 3. 1). Of the 51 PIRE projects with at least one journal article matched to Web of Science 
data, 1 1 (22 percent) produced 51 or more articles, with two of these producing more than 80. Most 
projects (17 projects; 33 percent) produced up to 10 peer reviewed journal articles (i.e., matched to 
Web of Science records); another 23 projects (45 percent) produced between 11 and 50 articles. 
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Exhibit 3.2: Distribution of Number of Journai Articies per PiRE Project (N=51 projects) 

20 


15 


projects 


5 


0 

1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 

Number of articles per PIRE project (N=51) 

Sources: Annual reports submitted by Pis to NSF; Thomson Reuters ’ Web of Science 

Notes: Of the 59 PIRE projects in the study, 8 had no journal articles identified in the Web of Science. These 8 projects 
include 6 (5 from the 2012 PIRE cohort, 1 from the 2010 cohort) that reported no journal publications by the time data were 
collected and 2 projects (1 from the 2012 cohort, 1 from the 2007 cohort) where none of the reported publications matched 
records in the Web of Science. 



The distribution of the quality of publications, as measured by the field normalized citation impact, 
also illustrates the wide variation across PIRE-produced articles and the highly skewed nature of the 
data (Exhibit 3.3). 

Exhibit 3.3: Distribution of the Fieid NCi of PiRE-Produced Pubiications (N=1,399 articies) 



0 >0 to <1 1 to <2 2 to <3 3 to <4 4 to <5 5+ 

Field Normalized Citation Impact (Field NCI) 


Sources: Annual reports submitted by Pis to NSF; Thomson Reuters ’ Web of Science 
Note: There were 60 matched articles with no reported NCI in the Web of Science. 
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3.2 PIRE and Comparison Projects’ Publications 

Next, we used comparative descriptive analyses to look for differences between PIRE and 
comparison projects’ research outcomes, with OLS regression models that account for the matched 
pairs to test for statistically significance of any observed differences. (Note that these analyses do not 
permit conclusions about the causes of any observed differences between research outcomes for the 
two groups.) 

In addition to comparing citation impacts and the representation of foreign coauthors for these two 
groups’ publications, we also present data illustrating the “global reach” of PIRE and comparison 
projects’ publications, including the extent to which authors at international institutions were citing 
PIRE research and the extent to which PIRE publications were citing research that involved 
international researchers. 

There were no statistically significant differences between PIRE and the comparison group for the 
quantity or quality of research produced (Exhibit 3.4). However, PIRE publications had a 
significantly higher percentage of foreign contributions, measured either as the percentage of foreign 
institutions represented among the authors per paper (PIRE articles had, on average, 35 percent 
foreign institutions versus 17 percent for comparison group publications) or measured as the 
percentage of foreign authors per paper (PIRE articles had an average of 29 percent foreign authors 
versus an average of 13 percent for comparison group). 

An analysis of citations of PIRE and comparison group articles shows that papers citing PIRE 
projects’ research articles have a higher percentage (66 percent), on average, of foreign institutions 
per paper than do papers that cite comparison projects’ articles (54 percent; Exhibit 3.4). Also, PIRE 
articles cite papers with a higher percentage of foreign institutions than on papers that comparison 
projects’ articles cite. Taken together, these two findings suggest that PIRE research is more widely 
disseminated among international institutions (and their researchers) than is research produced by the 
comparison group, and PIRE researchers are more familiar with research that has high international 
representation. 


As discussed in Chapter 2, for publications prior to 2009, Thomson Reuters cannot distinguish individual 
authors’ affiliations but can only count the number of distinct institutions with which authors are affiliated. 


Abt Associates 


Evaluation of NSF’s PIRE program: Volume 1 1 pg. 24 



Research Outcomes 


Exhibit 3.4: Comparison of Bibiiometric Measures (Regression- Adjusted) for PiRE and Comparison Group Articies (N=45 matched 
project pairs) 



Sources: Annual reports submitted by Pis to NSF; Thomson Reuters ’ Web of Science 
Notes: 

* Across papers citing one or more PIRE or comparison project-produced publications, the percentage of authors’ institutions per citing publication that were located outside the 
U.S.. 

^ Across papers that were cited by one or more PIRE or comparison project-produced publications, the percentage of authors’ institutions per cited publication that were located 
outside the U.S.. 
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3.3 PIRE Participants’ Publications 

Exhibits 3.5, 3.6, and 3.7 show the quality and quantity of post-onset publications co-authored by Pis, 
postdocs, and graduate students involved in PIRE, as well as the percentage of co-authors’ institutions 
per paper that are located outside the U.S. On average. Pis produced over 15 journal articles, while 
postdocs produced over 5 and graduate students produced more than one. These publications were, on 
average, more frequently cited than the average research article as measured by both field NCI and 
journal NCI. The percentage of foreign institutions per publication shows that for PIRE Pis’ 
publications, on average, just under one -third of co-authors’ institutions are foreign (31.8 percent). 


Exhibit 3.5: PiRE Principai investigators’ (N=307) Post-Onset Pubiications 


Bibliometric 

Indicator 

Nof 

articles 

Minimum 

Q1 

Median 

Q3 

percentile 

Maximum 

Mean 

SD 

Total number of 
post-onset articles 
(per PI) 

4,776 

0 

2 

6 

14 

59 

666 

15.56 

44.73 

Field NCI 

4,129 

0.00 

0.34 

1.17 

2.60 

9.08 

307.90 

2.72 

9.76 

Journal NCI (ratio 
observed to 
expected citations) 

4,146 

0.00 

0.28 

0.81 

1.76 

5.37 

90.20 

1.54 

3.20 

Percentage of 
foreign institutions 

4,759 

0.0% 

0.0% 

33.3% 

58.7% 

75.4% 

100.0% 

31.8% 

31.6% 


Exhibit 3.6: PiRE Postdoctorai Researchers’ (N=207) Post-Onset Pubiications 


Bibliometric 

Indicator 

Nof 

articles 

Minimum 

Q1 

Median 

Q3 

SS'" 

percentile 

Maximum 

Mean 

SD 

Total number of 
post-onset articles 
(per postdoc) 

1,113 

0 

0 

2 

6 

22 

73 

5.38 

10.11 

Field NCI 

938 

0.00 

0.26 

1.23 

2.85 

9.75 

307.90 

3.58 

13.67 

Journal NCI (ratio 
observed to 
expected citations) 

939 

0.00 

0.27 

1.00 

2.15 

7.03 

63.35 

2.00 

3.68 

Percentage of 
foreign institutions 

1,111 

0.0% 

0.0% 

42.9% 

75.0% 

100.0% 

100.0% 

44.1% 

35.9% 
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Exhibit 3.7: PiRE Graduate Students’ (n=467) Post-Onset Pubiications 


Bibliometric 

Indicator 

Nof 

articles 

Minimum 

Q1 

Median 

Q3 

SS'" 

percentile 

Maximum 

Mean 

SD 

Total number of 
post-onset articles 
(per student) 

807 

0 

0 

0 

1 

7 

180 

1.73 

8.73 

Field NCI 

667 

0.00 

0.23 

1.14 

2.86 

10.34 

280.67 

3.08 

11.84 

Journal NCI (ratio 
observed to 
expected citations) 

667 

0.00 

0.18 

0.86 

1.99 

5.82 

90.20 

1.79 

4.38 

Percentage of 
foreign institutions 

807 

0.0% 

0.0% 

50.0% 

73.7% 

80.0% 

100.0% 

38.6% 

33.8% 


Sources: Web of Science and custom calculations by Thomson Reuters. 


Notes: 

’ Articles on which a given participant is an author may include publications that did not result from participation in the 
PIRE project. For participants without a reported participation start date (i.e., from the study survey), the PIRE award 
effective date was used as participation start date. 


Across PIRE participants, the distribution of the number of journal articles is positively skewed, such 
that a small minority of participants produced a large number of post-onset publications with most 
participants producing a smaller total number of post-onset publications (Exhibit 3.8 presents a more 
fine-grained illustration of the skewed nature of the distribution than Exhibit 3.7). 


Exhibit 3.8: Distribution of PiRE Participants’ Number of Post-Onset Articies 


Number of 
participants 



Number of post-onset published journal articles 


Similarly, the distribution of the quality of these PIRE participants’ publications, as measured by the 
field normalized citation impact, also illustrates wide variation across articles and the highly skewed 
nature of the data (Exhibit 3.9). A small percentage of post-onset publications are more highly cited 
articles than average (e.g., field NCIs of 5 or higher) with a large proportion of publications having 
about a number of citations approximately equal to the world baseline average (i.e., field NCIs 
between 0 and 2). 
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Exhibit 3.9: Distribution of the Fieid NCI for PIRE Participants’ Post-Onset Articles 



3.4 PIRE versus Comparison Participants’ Publications 

Research outcomes were also explored at the participant level for PIRE and comparison project 
participants. We first present a descriptive comparative analyses of the percentage of foreign 
contrihutions to PIRE and comparison participants’ post-onset publications, using OLS regression 
models that control for the percentage of foreign contrihutions to pre-onset publications. (Note that 
these analyses do not permit conclusions about the causes of any observed differences between 
research outcomes for the two groups.) 


We then present findings from an impact analysis. Data from articles published before and after the 
onset of participation from both the PIRE and comparison participants permitted a quasi-experimental 
“difference-in-differences” analysis to estimate the impact of PIRE on participants’ research 
outcomes. (See Chapter 2: Methodology for further details.) An impact analysis that used random 
assignment would support the conclusion that any differences in PIRE and the comparison group’s 
outcomes were the result of participation in PIRE. However, because the impact analyses we used 
were quasi-experimental rather than experimental, attributions of differences in research outcomes to 
the PIRE program must acknowledge the possibility of alternative explanations. 


Percentage of foreign institutions on participants’ post-onset publications 
Eirst we examined the foreign contributions to research as measured by the percentage of foreign 
institutions on post-onset publications, controlling for the percentage on pre-onset publications. PIRE 
Pis’ post-onset publications had, on average, a statistically significant higher percentage of non-U.S. 
institutional affiliations (i.e., a proxy measure of the percentage of foreign co-authors) than 
comparison Pis’ post-onset publications. While postdocs’ and graduate students’ publications also 
had higher percentages of non-U.S. institutional affiliations than publications by their comparison 
project peers, these differences were not statistically significant. 
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Exhibit 3.10: Regression- Ad justed Mean Percentage of Non-U.S. institutionai Affiiiations for 
Post-Onset Pubiications 


Respondent Group 

Adjusted Mean Percentage of Non-U.S. Institutional Affiliations for 
Post-Onset Publications = 

PIRE 

Comparison 

Difference 
(PIRE - 
Comparison) 

Standard 

Error 

p-Value 

PIs^’ 

15.5 

9.7 

5.8 

1.66 

.001** 

Postdocs'^ 

15.5 

12.4 

3.1 

5.39 

0.569 

Graduate students'^ 

17.4 

7.3 

10.1 

8.44 

.236 


Sources: Web of Science. 


Notes: 

■* The regression model controlled for the pre-onset percentage of foreign institutions per publication per participant and 
was calculated for the subset of participants who had at least one pre-participation publication and at least one post- 
participation publication. The comparative analysis was an ordinary least squares regression with a group indicator 
(PIRE=1, Comparison=0) and dummy variables representing the matched pairs of projects. A pair of projects was 
included in the analysis sample only if each project within a pair had at least one participant with pre- and post-onset 
publications. 

FIRE N=204; Comparison N=168 in 54 matched project pairs 
' FIRE N=48; Comparison N=55 in 34 matched project pairs 
FIRE N=36; Comparison N=56 in 45 matched project pairs 

Impact of PIRE on productivity and field normalized citation impact 

Next we examined the impaet of PIRE on partieipants’ annual number of publications (i.e., the 
number of publications produced per year) and average field normalized citations per year (across 
articles published within a given year). Exhibit 3.11 summarizes the outcomes of PIRE versus 
comparison project participants for Pis, postdoctoral researchers, and graduate students. We found no 
significant differences in annual number of publications among Pis. However, among postdocs, PIRE 
participants had a larger number of average post-award annual publications (.46 more publications 
per year) and these publications had a larger impact (cited .57 times more often). Among graduate 
students, PIRE participants also had a higher average number of annual post-award publications than 
the comparison group (.23 more publications per year). 
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Exhibit 3.11 : impact of PiRE on Pis’, Postdocs’ and Graduate Students’ Research Outcomes^ 


Outcome 

PIRE 

Comparison 




Pre 

Mean 

Post 

Mean 

(Post - 
Pre) 

Pre 

Mean 

Post 

Mean 

(Post - 
Pre) 

Impact 

Estimate 

Standard 

Error 

p-value 


Pis'* 


Annual publications^ 

2.02 

2.46 

0.44 

1.71 

2.06 

0.35 

0.09 

0.13 

0.486 

Field NCI per year^ 

1.28 

1.60 

0.32 

1.66 

1.94 

0.28 

0.04 

0.11 

0.708 

Postdocs^ 

Annual publications 

0.33 

1.38 

1.04 

0.39 

0.98 

0.59 

0.46 

0.11 

<.0001 

Field NCI per year 

0.43 

2.02 

1.58 

0.65 

1.66 

1.01 

0.57 

0.22 

0.010 

Graduate Students^ 

Annual publications 

0.15 

0.78 

0.63 

0.11 

0.52 

0.40 

0.23 

0.09 

0.014 

Field NCI per year 

0.51 

1.15 

0.64 

0.43 

0.99 

0.56 

0.08 

0.13 

0.549 


Sources: Web of Science and custom calculations by Thomson Reuters for NCI per year. 


Notes: 

’ We conducted a separate analysis for each respondent group (Pis, postdocs, graduate students). For each analysis, the 
impact model estimated the PIRE-comparison difference between the pre- and post-onset outcome measure for each 
group controlling for the matched pairs of projects. For each respondent group, we restricted analysis sample to a 
subset for which baseline equivalence on first year of participation, pre-onset annual publications and field NCI per 
year. For further details, see Volume 2. 

^ Annual publications refers to the number of publications published by a participant per year. The impact analysis 
method (CITS) requires yearly data rather than a single total number of publications. 

^ The field NCI per year for each respondent group is the average NCI across a participant’s articles published in a given 
year across the participants. 

Pis: PIRE N=256, Comparison N=I99 in 54 matched project pairs 

^ Postdocs: PIRE N=I02, Comparison N=I 18 in 34 matched project pairs 

® Graduate students’ annual publications: PIRE N=I72, Comparison N=296 in 45 matched project pairs; Graduate 
students’ field NCI: PIRE N=170, Comparison N=287 in 45 matched project pairs. 
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4. Participant Experiences 


This chapter provides information about participants’ experiences in PIRE. To investigate the 
progress of the program toward its goals, we paid particular attention to the opportunities for 
international research collaborations, the experiences that develop research capacity and global 
perspectives, and the relationships between U.S. and foreign researchers. Specifically, the findings 
address the following research questions: 

• RQ3b. What are the program experiences and educational or career outcomes of PIRE Pis, 
postdoctoral researchers, and graduate and undergraduate student participants? 

• RQ4b. How do the program experiences and educational or career outcomes of these PIRE 
participants compare to similar participants in this comparison group of NSF-funded projects? 

• RQ 6. What are the program experiences of foreign senior investigative partners in PIRE and how 
do they perceive the effects of PIRE on their research and educational practices and those of their 
institutions? 

Data in this chapter come from the surveys completed by PIRE Pis, postdocs, graduate students, 
undergraduate students and foreign investigators, and the surveys completed by comparison group 
project Pis, postdocs, and graduate students.’® We first present findings for PIRE Pis, postdocs, and 
graduate students and compare these participants’ experiences to that of their peers in comparison 
projects. We then describe the experiences of undergraduate students who participated in PIRE and 
traveled abroad and the experiences of foreign senior investigators in PIRE. 

• A greater proportion of Pis, postdoctoral researchers, and graduate students in PIRE projects 
traveled outside the U.S. than participants in comparison projects. For each group, the percent 
was well over three -fifths among PIRE participants and less than one -quarter among participants 
in comparison projects. Further, PIRE postdocs and graduate students were more likely than their 
counterparts in comparison projects to have traveled abroad to conduct research. 

• Collaboration with individuals within the U.S. was commonly reported by both PIRE and 
comparison Pis, postdocs and graduate students; however, PIRE participants were more likely to 
collaborate with individuals based primarily outside the U.S. 

• PIRE Pis and graduate students were more likely to report maintaining these international 
collaborations than comparison project participants, and commonly these relationships continued 
to focus on research-related exchanges. 

• PIRE postdocs, graduate students, and undergraduates would recommend their peers engage in 
international research and education because of the benefits associated with working with and 
learning from international collaborators, cultural and social exchange, enhanced professional 
networking, and skills and knowledge developed as a researcher. Challenges were not commonly 
mentioned by respondents. 

• Undergraduates who traveled outside the U.S. for PIRE-related research were more likely to have 
collaborated with foreign faculty, senior- or postdoctoral-level researchers or graduate students 
rather than with foreign undergraduate students. 


Because we did not field surveys of undergraduate students or foreign investigators involved in comparison 
group projects, we present descriptive analyses only for these groups. 
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• Among undergraduate respondents who traveled outside the U.S. for PIRE-related research, the 
majority reported that their participation in the project yielded a range of personal benefits . Most 
commonly reported among these was an understanding of how scientists work on real problems 
(81 percent), followed by tolerance for obstacles faced in the research process (71 percent), self- 
confidence in their field of study (69 percent), and clarification of a career path (67 percent). 

• Foreign investigators were less likely than U.S. -based investigators to have traveled 
internationally during their participation in the PIRE project (48 percent). Among those who 
traveled abroad, the majority traveled to the U.S. (78 percent). 

• Among most foreign investigators, PIRE was not the first international research collaboration 
with a U.S. -based partner; 82 percent reported working with a U.S. -based individual on research 
prior to their participation in the PIRE project. 

4.1 Pis, Postdocs, and Graduate Students 

4.1.1 Travel Outside the U.S. During the Project 

A greater proportion of Pis, postdoctoral researchers, and graduate students in PIRE projects traveled 
outside the U.S. than the proportion of these participants in comparison projects (Exhibit 4.1). Just 
under one-fourth of Pis (24 percent) in the comparison group reported traveling abroad for their 
project, compared to more than four-fifths (84 percent) of PIRE Pis. The majority of PIRE postdocs 
and graduate students traveled abroad (60 percent and 76 percent, respectively) compared to just 9 
percent of comparison project postdocs and 17 percent of graduate students participating in a 
comparison group project. These differences were statistically significant for each group (Pis, 
postdocs, and graduate students). 

Exhibit 4.1 : Percent of Participants Who Traveled Outside the U.S. For the Project 

100 

80 
60 

Percent 

40 
20 
0 



Sources: Principal Investigator Survey, Item C2; Postdoc Survey, Item B2; Graduate Student Survey, Item B3. 

Notes: 

Pis: PIRE N=157, Comparison N=I 14; Postdocs: PIRE N=1 14, Comparison N=1 15; Graduate students: PIRE N=262, 
Comparison N=33I. 
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On average, PIRE Pis made slightly more frequent but shorter trips abroad (six trips lasting about 2 
weeks on average) than Pis in comparison projects (four trips of about 3 weeks in duration; see 
Exhibit 4.2). Both PIRE and comparison group postdocs reported an average of three trips abroad, but 
the average duration of a trip for PIRE postdocs was nearly three times as long ( 14 weeks) than for 
comparison group postdocs (5 weeks). Similarly, PIRE graduate students made the same number of 
trips on average as comparison graduate students but the PIRE graduate students’ trips lasted more 
than twice as long (11 weeks), on average, than those of comparison graduate students’ (5 weeks). 


Abt Associates 


Evaluation of NSF’s PIRE program: Volume 1 1 pg. 33 


Participant Experiences 


Exhibit 4.2: Characteristics of Participants’ Travei Abroad for the Project 


Of Those Who Traveled Abroad: 

PI 

Postdoc 

Graduate Student 

PIRE 

Comparison 

PIRE 

Comparison 

PIRE 

Com 

parison 

N 

Mean 

N 

Mean 

N 

Mean 

N 

Mean 

N 

Mean 

N 

Mean 

Average number of trips abroad^ 

140 

6 

24 

4 

75 

3 

8 

3 

213 

2 

38 

2 

Average duration of trips abroad, in 
weeks^ 

140 

2.3 

24 

2.9 

71 

13.9 

5 

4.8 

213 

11.4 

38 

5.0 

Longest trip abroad, in weeks^ 

140 

4.5 

24 

4.1 

72 

15.3 

5 

5.8 

212 

13.4 

37 

6.4 


Sources: 

Average number of trips: Principal Investigator Survey, Item CSa; Postdoc Survey, Item B6a; Graduate Student Survey, Item B9a. 

Average duration: Principal Investigator Survey, Items CSb and C5d; Postdoc Survey, Items B6b and B6d; Graduate Student Survey, Item B9b and B9d. This captures the average 
duration of single and multiple trips. 

Longest trip: Principal Investigator Survey, Items C5c and C5d; Postdoc Survey, Items B6c and B6d; Graduate Student Survey, Items B9c and B9d. This captures the duration of 
the longest trip abroad for single and multiple trips. 

Notes: 

' Number of trips: PIRE N=140, missing=l, Comparison N=24. Postdocs: PIRE N=75, missing=3, Comparison N=8. Graduate students: PIRE N=213, missing=3. Comparison 
N=38, missing=3. 

^ Average duration: Pis: PIRE N=140, missing=l, Comparison N=24; Postdocs: PIRE N=7I, missing=7. Comparison N=5, missing=3; Graduate students: PIRE N=2I3, 
missing=3. Comparison N=38, missing=3. 

^ Longest trip: Pis: PIRE N=140, missing=I, Comparison N=24; Postdocs: PIRE N=72, missing=6, Comparison N=5, missing=3; Graduate students: PIRE N=212, missing=4; 
Comparison N=37, missing=4. 
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Greater proportions of PIRE postdoctoral and graduate students traveled outside the U.S. for research 
purposes and a greater percentage of PIRE Pis traveled abroad for “research or educational activities” 
related to the project, compared to their peers in comparison group projects (Exhibit 4.3). These 
differences in reasons for traveling abroad were statistically significant. 


Exhibit 4.3: Regression-Adjusted Mean Percentage of PiRE and Comparison Participants Who 
Traveied Abroad for Stated Purpose 


Respondent 

Type 

Traveled Abroad for 

Mean Percent^ 

Difference 

Standard 

Error 

p-Value 

Statistical 

Significance 

PIRE 

Comparison 

PIS2 

Research or 
educational activities 

84.0 

23.7 

60.4 


<.01 

** 

Postdocs^ 

Conduct research 

97.8 

42.8 

55.0 

16.5 

0.002 

*** 

Graduate 

students 

Conduct research 

94.2 

79.7 

14.4 

5.8 

0.014 

* 


Sources: Principal Investigator Survey, Item C2; Postdoc Survey, Item B4; Graduate Student Survey, Item B5. 


Notes: 

’ The regression model controlled for the fact that participants were within matched pairs of projects (N=55 pairs). 

^ Pis, PIRE N=157, Comparison N=1 14. Because Pis were asked if they traveled abroad “to conduct research or 

educational activities (other than travel solely to attend a professional conference)” it is not possible to disambiguate 
“research” from “educational” purposes for this participant group. 

^ Postdocs, PIRE N=72, Comparison N=8; Graduate Students: PIRE N=195, Comparison N=42. Postdocs and graduate 
students were asked to identify the purposes for which they traveled abroad as part of the project. Those who indicated 
“conduct research” are shown. 

Exhibit 4.4 shows a broader range of activities for which postdoctoral and graduate student 
participants traveled abroad. In addition to traveling outside the U.S. to conduct research, the next 
most frequently reported activity abroad for PIRE postdocs was attending planning meetings or 
workshops; in contrast, the next most frequently reported activity abroad for comparison postdocs 
was attending a professional conference. No comparison postdocs taught a course while abroad, 
compared to 17 percent of PIRE postdocs. Eive percent of comparison graduate students took classes 
while abroad, compared to nearly one-fifth of PIRE graduate students who traveled abroad. 
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Exhibit 4.4: Reasons Postdoctorai and Graduate Student Participants Reported for Travei Abroad for the Project 



Postdocs 

Graduate Students 


PIRE 

Comparison 

PIRE 

Comparison 


N 

Percent 

N 

Percent 

N 

Percent 

N 

Percent 

Conduct research^ 

72 

94% 

4 

50% 

201 

93% 

33 

79% 

Attend planning meetings or workshops for upcoming research^ 

28 

45% 

2 

25% 

80 

37% 

13 

31% 

Attend a professional conference^ 

28 

36% 

4 

50% 

62 

29% 

7 

17% 

Other— specify'* 

14 

18% 

2 

25% 

19 

9% 

5 

12% 

Teach and/or prepare curriculum for a course^ 

35 

17% 

0 

0% 

na 

na 

na 

na 

Take classes at a university or other educational institution® 

3 

4% 

0 

0% 

41 

19% 

2 

5% 

Participate in an internship'' 

na 

na 

na 

na 

11 

5% 

2 

5% 

None of the above 

0 

0% 

0 

0% 

0 

0% 

0 

0% 


Sources: Postdoctoral Survey, Item B4; Graduate Student survey, Item B5. 


Notes: 

' Conduct research: Postdocs: PIRE N=77, missing=l, Comparison N=8; Graduate students: PIRE N=216, Comparison N=42. 

^ Attend planning meetings/workshops: Postdocs: PIRE N=77, missing=l, Comparison N=8; Graduate students: PIRE N=216, Comparison N=42. 

^ Attend professional conference: Postdocs: PIRE N=77, missing=l. Comparison N=8; Graduate students: PIRE N=216, Comparison N=42. 

Other: Postdocs: PIRE N=77, missing=l, Comparison N=8; Graduate students: PIRE N=216, Comparison N=42. 

^ Teach/prepare for course: Postdocs: PIRE N=77, missing=l. Comparison N=8. Graduate students were not asked if they traveled abroad to teach or prepare curriculum for a 
course. 

^ Take classes at university/educational institution: Postdocs: PIRE N=77, missing=I, Comparison N=8; Graduate students: PIRE N=216, Comparison N=42. 

’’ Participate in internship: Graduate students: PIRE N=2I6, Comparison N=42. Postdoctoral participants were not asked if they traveled abroad to participate in an internship 
* None of the above: Postdocs: PIRE N=77, missing=l. Comparison N=8; Graduate students: PIRE N=216, Comparison N=41, missing=I. 
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4.1.2 Collaborations 

Participants reported on collaborations during and subsequent to their participation in the project with 
individuals who were primarily based in and outside of the U.S. 

Collaborations during participation in the project 

The overwhelming majority of both PIRE and comparison participants had collaborated during the 
project with individuals within the U.S. (Exhibit 4.5). Across the PI, postdoctoral and graduate 
student respondents, a larger proportion of PIRE participants versus comparison project participants 
had collaborated with individuals based primarily outside the U.S. 

Exhibit 4.5: Percent of Participants Who Collaborated During the Project with Individuals 
Based Within^ and Outside the U.S.^ 

100 1 
80 - 

- 60 - 

0 ) 
o 

0 ) 

40 - 
20 - 
0 - 

Pis Postdocs Graduate students 

Sources: Principal Investigator Survey, Item Bla; Postdoc Survey, Item Bla; Graduate Student Survey, Item B2. 

Notes: 

1 Within U.S.: PIRE N=157, Comparison N=114; Postdocs: PIRE N=II7, Comparison N=1I6, missing=l; Graduate 
students: PIRE N=235, Comparison N=3I6. 

2 Outside the U.S.: Pis: PIRE N=157, Comparison N=1 12; Postdocs: PIRE N=1 17, Comparison N=1 16, missing=l; 
Graduate students: PIRE N=234, Comparison N=316. 

Collaborations subsequent to participation in the project 

A key outcome for the study is the extent to which collaborations with foreign colleagues continued 
after U.S. participants’ role in the project had concluded. The study collected information on these 
collaborations for PIRE and comparison Pis, postdocs, and graduate students, as well as descriptive 
data on the nature of these continued collaborations and any subsequent collaboration with foreign 
colleagues who were not part of the PIRE or comparison project teams. Only participants in expired 
projects (i.e., PIRE or comparison projects in which the official NSF award period had ended) and 
participants who reported that their contributions to the project had ended prior to the survey 
reference date were asked to report on these post-participation outcomes. 

Post-award (or post-participation), PIRE Pis and graduate students were significantly more likely to 
collaborate with non-U.S. -based colleagues who had been part of the reference award (Exhibit 4.6); 
there were no statistically significant differences among postdocs. These collaborations commonly 
involved discussing developments in their research field(s), planning research projects, conducting 
research, and sharing data or other resources (Exhibits 4.7). These types of research-related 
collaborative activities were reported by larger proportions of PIRE than comparison group 


97 99 


95**100 



I Within the U.S.: PIRE 
I Within the U.S.: Comparison 
I Outside theU.S.: PIRE 
I Outside the U.S.: Comparison 
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participants. This suggests that the continued collaborations went heyond simply encountering foreign 
colleagues at conferences. 


Exhibit 4.6: Percent of Participants (Regression Adjusted) Who Continued, After Participation 
in the Project, to Coiiaborate with a Non-U.S.-Based Coiieague Who Was a Member of the PiRE 
or Comparison Project’ 



Pis Postdocs Graduate 

Students 


■ PIRE 

■ Comparison 


Respondent group ^ 


Sources: Principal Investigator Survey, Item Bib; Postdoc Survey, Item Bib; Graduate Student Survey, Item B2a. 

Notes: 

’ Stepwise regression models controlled for the fact that participants were within matched pairs of projects (N=55 pairs); 
other co variates were excluded in the PI and postdoc models; for graduate students only, the regression model also 
included a covariate for graduate field of study. Non-U. S. -based colleagues was restricted to include only project 
members who “were based outside the U.S.” and who “still are” based outside the U.S., to the best of the respondent’s 
knowledge at the time of the survey. When the definition of non-U.S. -based colleague is expanded to include both (a) 
project members who were and still are based outside the U.S. and (b) project members who were based outside the 
U.S. but who “are now” based primarily within the U.S., the results change by less than two percentage points for PIRE 
or comparison participants. 

^ Pis, PIRE N=62, Comparison N=48; Postdocs, PIRE N=60, Comparison N=83; Graduate Students: PIRE N=149, 
Comparison N=219. 

^ Asterisks indicate that the difference between PIRE and comparison groups is statistically significant: *p<.05, **p<.01 


PIRE Pis and graduate student participants also were significantly more likely to collahorate post- 
award (or post-participation) with non-U.S. -based colleagues who had not been part of the reference 
award (Exhibit 4.8); again, there were no statistically significant differences among postdocs who had 
participated in a PIRE or comparison project. 
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Exhibit 4.7: Percent Reporting Seiected Types of Coiiaborative Activities with Non-U.S.-Based 
Coiieague from the Project Team, After Participation in the Project Had Conciuded 


Type of Activity 

Pis 

Postdocs 

Graduate Students 

PIRE 

Comparison 

PIRE 

Comparison 

PIRE 

Comparison 

Discussed developments in 
research field(s) with a foreign 
partner 

91% 

62% 

84% 

75% 

76% 

64% 

Planned one or more research 
projects with a foreign partner 

85% 

52% 

74% 

30% 

58% 

24% 

Conducted one or more 
research projects with a foreign 
partner 

82% 

52% 

48% 

25% 

64% 

45% 

Shared data or other resources 
with a foreign partner 

82% 

57% 

71% 

40% 

71% 

55% 

Traveled outside the U.S. to 
work with one or more foreign 
partners 

76% 

43% 

58% 

20% 

69% 

21% 

Met with one or more foreign 
partners at conferences outside 
the U.S. 

67% 

52% 

52% 

35% 

51% 

15% 

A foreign partner has traveled 
to the U.S. to work with me 

55% 

52% 

35% 

15% 

36% 

27% 

A former U.S.-based partner 
has (or has had) a position at 
an institution outside the U.S. 

29% 

29% 

29% 

30% 

13% 

36% 

A former foreign partner has (or 
has had) a position at an 
institution in the U.S. 

25% 

14% 

32% 

25% 

18% 

15% 

Have (or have had) a position 
at an institution outside the U.S. 

16% 

5% 

39% 

25% 

16% 

9% 

None of the Above 

0% 

0% 

0% 

15% 

4% 

9% 


Sources: Principal Investigator Survey, Item Bid; Postdoctoral Survey, Item Bid; Graduate Student Survey, Item B2c. 
Notes: Pis: PIRE N=55, Comparison N=21; Postdocs: PIRE N=31 ; Comparison N=20, missing=l; Graduate students: PIRE 
N=55, Comparison N=33. 
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Exhibit 4.8: Percent of Participants (Regression- Adjusted) Who Coiiaborated, After 
Participation in the Project, with a Non-U.S.-Based Coiieague Who Was Not a Member of the 
PiRE or Comparison Project 



Sources: Principal Investigator Survey, Item Ble; Postdoc Survey, Item Ble; Graduate Student Survey, Item B2d. 

Notes: 

* Stepwise regression models controlled for the fact that participants were within matched pairs of projects (N=55 pairs); 
other covariates were excluded in the PI and postdoc models; for graduate students only, the regression model also 
included a covariate for graduate field of study. 

^ Pis, PIRE N=62, Comparison N=49; Postdocs, PIRE N=69, Comparison N=90; Graduate Students: PIRE N=I50, 
Comparison N=2I9. 

^ Asterisks indicate that the difference between PIRE and comparison groups is statistically significant: *p<.05, **p<.01 

4.1.3 Educational and Career Outcomes^^ 

We asked Pis, postdocs and graduate students to report on a handful of other post-participation career 
or educational outcomes. Because these groups were at different points in their careers (or educational 
trajectories), these items differed somewhat between groups. 

A majority of Pis from both PIRE (68 percent) and comparison projects (67 percent) reported that 
their NSF award had helped them leverage other funds to support their research (Exhibit 4.9). A 
larger proportion of comparison postdocs (46 percent) than PIRE postdocs (26 percent) indicated that 
the project experience helped them win grant funding for their research, whereas a higher percentage 
of PIRE graduate students than comparison project graduate participants reported this effect. None of 
these differences were statistically significant. 


Volume 2 includes findings from a secondary set of analyses comparing PIRE postdoctoral, graduate and 
undergraduate participants’ employment outcomes to national trends using data from the Survey of 
Doctorate Recipients and the National Survey of Recent College Graduates. 
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Exhibit 4.9: Percent (Regression-Adjusted) of PiRE and Comparison Participants Reporting 
that the Project Heiped Secure a Benefit^ 



PIRE 

Comparison 

Difference 

p-value 

The project helped secure funds to support research 

Principal Investigators^ 

68% 

67% 

1.4 

0.830 

Postdoctoral researchers^ 

26% 

46% 

-20.5 

0.125 

Graduate students^ 

40% 

31% 

9 

0.173 

The project helped secure a promotion or first choice of full-time position' 

Principal Investigators^ 

34% 

32% 

2.1 

0.743 

Postdoctoral researchers^ 

65% 

68%% 

-2.6 

0.815 

Graduate students^ 

59% 

53 

6.3 

0.402 


Sources: Principal Investigator Survey, Item D7b; Postdoc Survey, Item E5 ; Graduate Student Survey, Item E6. 


Notes: 

’ Stepwise regression models controlled for the fact that participants were within matched pairs of projects (N=55 pairs); 
other covariates were excluded in the PI and postdoc models; for graduate students only, the regression model also 
included a covariate for graduate field of study. 

^ Pis were asked whether the project (a) provided an opportunity to leverage other funds to support their research and (h) 
helped them secure a promotion. PIRE N=157, Comparison N=1 14. 

^ Postdocs and graduate students were asked whether the project (a) helped them win grant funding to support their 

research and (h) helped them receive a full-time position that was their top choice. Postdocs, PIRE N=49, Comparison 
N=58. Graduate students, PIRE N=I 1 1, Comparison N=147. 

A greater percentage of comparison project (former) graduate students than PIRE (former) graduate 
students who completed a doctorate reported that they subsequently held a postdoctoral appointment, 
hut among those who did, a greater percentage of PIRE’ s former graduate participants than the 
comparison group had a postdoctoral appointment outside the U.S. Neither difference was statistically 
significant. 


Exhibit 4.10: Percent (Regression-Adjusted) of PiRE and Comparison Graduate Students Who 
Heid a Postdoctorai Position After Compieting Their Doctorate’ 


Former Graduate Student Participants 

PIRE 

Comparison 

Difference 

p-Value 

Ever had a postdoctoral appointmenP 

60% 

70% 

-9.4 

0.206 

Had a postdoctoral appointment outside the 
U.S.3 

31% 

17% 

13.9 

0.089 


Sources: Graduate Student Survey, Items EIc, Eld. 
Notes: 


Stepwise regression models controlled for the fact that participants were within matched pairs of projects (N=55 pairs) 
and included a covariate for graduate field of study. 

^ PIRE N=100, Comparison N=137. 

^ PIRE N=59, Comparison N=88. 

One potential effect of winning an NSE grant is that it could raise a Pi’s academic or public profile 
and lead either to unsolicited job offers or opportunities to move from one institution to one perceived 
as more prestigious. Few PIRE or comparison Pis reported that the project had provided a means to 
move to a more prestigious institution, although more than 40 percent of PIRE and comparison Pis 
had received unsolicited job offers since the start of their award. There were no significant differences 
for either outcome. 
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Exhibit 4.1 1 : Percent (Regression-Adjusted) of PiRE and Comparison Pis Who Reported that 
the Project Led to Job Opportunities or Offers^ 


Principal Investigators 

PIRE 

Comparison 

Difference 

p-Value 

Provided an opportunity to move to a more prestigious 
institution^ 

12% 

7% 

5 

0.203 

Since the start of the project have received unsolicited offers of 
employment 

42% 

42% 

-0.1 

0.983 


Sources: Principal Investigator Survey, Items D7b and D8. 
Notes: 


Stepwise regression models controlled for the fact that participants were within matched pairs of projects (N=55 pairs). 
PIRE N=157, Comparison N=1 14. The survey displayed the following mouse-over definition: “An unsolicited offer of 
employment is one that occurs independently of any action you initiated to seek different employment.” 

4.1.4 Perceived Benefits and Chaiienges 

Overwhelmingly, both postdocs and graduates students would recommend that their peers participate 
in international research and evaluation activities (Exhibit 4. 12). In describing why they would 
recommend that others participate in international research or educational activities, postdocs’ and 
graduate students’ top benefits cited included working with and learning from international 
collaborators, cultural and social exchange, enhanced professional networking, and skills and 
knowledge developed as a researcher. The experience working with and learning from international 
collaborators was also most commonly selected most beneficial aspect of their participation. 


Exhibit 4.12: Respondent Levei of Recommendation to Academic Peers on Participating in 
internationai Research or Educationai Activities 


Level of Recommendation 

Postdocs 

Graduate Students 

N 

% 

N 

% 

Strongly recommend 

34 

81% 

126 

82% 

Recommend 

7 

17% 

18 

12% 

Recommend with reservations 

1 

2% 

8 

5% 

Not recommend 

0 

0% 

2 

1% 


Sources: Postdoc Survey, Item E6; Graduate Student Survey, Item E7. 
Notes: Postdocs, N=42, missing=4. Graduate students, N=154, missing=5. 


Participants agreed that their research had benefited in a variety of ways from access to resources 
available through the non-U.S. locations and partnerships (Exhibit 4.13). Challenges were not 
commonly reported by respondents. We asked each group whether bureaucratic challenges or 
language differences in a foreign country hindered the research: less than one-third of the Pis, 
postdocs or graduate students agreed or strongly agreed that bureaucratic challenges hindered the 
progress of the research, and less than 20 percent of each group agreed or strongly agreed that 
language differences had hindered the research. When asked about other possible challenges, more 
than half of Pis cited no other challenges and more than 60 percent of postdoctoral and graduate 
student participants cited no other challenges (Exhibit 4.14). 
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Exhibit 4.13: Benefits and Chaiienges to Research 


Benefit/Challenge 

Principal Investigators 

Postdocs 

Graduate Students 

Strongly 
Agree or 
Agree 

Strongly 
Disagree or 
Disagree 

Not 

Appl. 

Strongly 
Agree or 
Agree 

Strongly 
Disagree or 
Disagree 

Not 

Appl. 

Strongly 
Agree or 
Agree 

Strongly 
Disagree or 
Disagree 

Not 

Appl. 

Research benefited from access to expertise of 
foreign partners in field 

94% 

4% 

2% 

96% 

3% 

1% 

86% 

5% 

9% 

Research benefited from access to data 
acquired in foreign locations 

85% 

6% 

9% 

84% 

6% 

9% 

76% 

7% 

17% 

Different cultural perspectives improved the 
quality of the research 

78% 

16% 

6% 

78% 

14% 

8% 

73% 

11% 

16% 

Research benefited from access to place- 
based phenomena that occur outside the U.S. 
(e.g., biological, geological, cultural 
phenomena) 

68% 

7% 

25% 

60% 

16% 

24% 

63% 

9% 

27% 

Research benefited from access to equipment 
or facilities in foreign locations 

66% 

12% 

21% 

63% 

20% 

17% 

65% 

13% 

22% 

Regulatory, bureaucratic or administrative 
procedures in a foreign country hindered the 
progress of the research 

31% 

60% 

9% 

22% 

62% 

16% 

29% 

49% 

23% 

Language differences hindered the progress of 
the research 

19% 

75% 

6% 

9% 

77% 

14% 

16% 

69% 

16% 


Sources: Principal Investigator Survey, Item D5; Postdoc Survey, Item C4; Graduate Student Survey, Item C4. 
Notes: Pis, N=170. Postdocs, N=95, missing=7. Graduate students, N=282-283, missing=2-3. 
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Exhibit 4.14: Other Chaiienges Experienced Whiie Participating in PiRE Project 


Challenges 

Principal 

Investigators 

Postdocs 

Graduate 

Students 

N 

% 

N 

% 

N 

% 

None 

89 

52% 

60 

63% 

182 

64% 

Key foreign partners were not as available as 1 
expected or did not contribute enough to the 
project^ 

33 

19% 

NA 

NA 

NA 

NA 

Inadequate access to data, specimens, facilities, 
equipment, other resources 

31 

18% 

12 

13% 

NA 

NA 

Funding expected from foreign (non-U.S.) sources 
did not materialize^ 

23 

14% 

NA 

NA 

NA 

NA 

Unexpected financial burdens^ 

NA 

NA 

8 

8% 

29 

10% 

Delays in getting publications written due to the 
cross-cultural nature of the team 

22 

13% 

7 

7% 

15 

5% 

Conflicts over authorship or credit for contributions 
to the research 

14 

8% 

9 

9% 

22 

8% 

Other 

12 

7% 

4 

4% 

13 

5% 

Bias from foreign partners based on my nationality, 
gender, race, ethnicity, religion, disability or sexual 
orientation 

5 

3% 

1 

1% 

12 

4% 

Not have enough information or guidance to 
prepare for the foreign component of the research^ 

NA 

NA 

6 

6% 

28 

10% 

Delays pursuing/completing goals of postdoctoral 
appointment or graduate degree'* 

NA 

NA 

11 

12% 

39 

14% 


Sources: Principal Investigator Survey, Item D6; Postdoc Survey, Item C5; Graduate Student Survey, Item C5. 


Notes: Pis, N=170. Postdocs, N=94-95, missing=7-8. Graduate students, N=284, missing=l. 

’ Postdocs and graduate students were not asked whether key foreign partners were not as available as expected or did 
not contribute enough to the project. 

2 Principal investigators were asked whether funding expected from foreign (non-U.S.) sources did not materialize. 
Postdoctoral researchers and graduate students were asked if they experienced unexpected financial burdens. 

3 Principal investigators were not asked whether they had enough information or guidance to prepare for the foreign 
component of the research. 

4 Principal investigators were not asked whether they experienced delays in pursuing/completing goals of their academic 
appointment. 


4.2 Undergraduate Students in PIRE 

Of the 59 PIRE awards, 50 included one or more undergraduate participants, from which we selected 
a sample to complete a study survey (Exhibit 4.15 shows a distribution of PIRE projects by the 
number of undergraduate participants). Their responses provide some insight into experiences of 
undergraduate students who participated in a PIRE project. 
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Exhibit 4.15: Distribution of PiRE Projects (N=59) by the Number of Undergraduate 
Participants 



0 1-5 6-10 11-20 21-30 30 + 

Number of Undergraduate Participants per Project 


Sources: Annual reports submitted by Pis to NSF. 

4.2.1 Travei Outside the U.S. During the Project 

Eighty-seven percent of undergraduate respondents reported that they had traveled abroad for the 
PIRE project to help conduct a research project, participate in an internship, attend a professional 
conference, and/or take classes or participate in educational activities abroad (Exhibit 4.16). The most 
commonly reported activity for which these undergraduate PIRE participants traveled abroad was 
research (90 percent), followed by participation in an internship (74 percent), attendance at a 
professional conference (65 percent), and participation in classes or education abroad (53 percent). 

On average, during their participation in PIRE, undergraduates whose travel was related to PIRE took 
one trip abroad and these trips lasted an average of eight weeks. The majority of undergraduates (83 
percent) reported that they had traveled with other U.S. -based peers, faculty, postdoctoral researchers 
or graduate students. 
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Exhibit 4.16: Characteristics of Undergraduates’ Travei Abroad for PiRE 


Travel Abroad 

Percent 

Traveled abroad for PIRE for one of four purposes 

Traveled abroad for the PIRE project to participate In: research project, Internship, professional 
conference and/or classes or educational program abroad^ 

87% 

Participated in activity/activities while abroad for the PIRE project: 

Help conduct an original research project^ 

90% 

Participate In an Internship^ 

74% 

Attend a professional conference'* 

65% 

Take classes at an educational Institution or participate In educational program abroad ® 

53% 

Traveled abroad with other U.S.-based individuals^ 

Other U.S.-based undergraduates 

83% 

U.S.-based faculty or postdoctoral researchers 

69% 

U.S.-based graduate students 

60% 

Of those who traveled abroad for the PIRE project: 

Mean 

Average number of trips abroad^ 

1.2 

Average duration of trips abroad. In weeks® 

8.0 

Longest trip abroad. In weeks® 

8.6 


Sources: Undergraduate Student Survey. 

Traveled abroad: Item BI. 

Participated in activity: Item B2. 

With other U.S. -based individuals: Item BI4. 

Average number of trips: Item BI3a. 

Average duration: Items BI3b and BI3d. This captures the average duration of single and multiple trips. 

Longest trip: Items BI3c and BI3d. This captures the duration of the longest trip abroad for single and multiple trips. 

Notes: 

’ Traveled abroad for the PIRE project: N=241 . Results show undergraduates who indicated that they had traveled 
outside the U.S. to (a) help with a research project, or take classes/participate in educational activities abroad, or 
participate in an internship or attend a professional conference and who indicated (Item B2) that the PIRE project was 
the reason for this activity/these activities abroad. Note that respondents who indicated “yes” or “not sure” that the 
activity was related to the PIRE project were included; although survey respondents had been identified as project 
participants in a Pi’s annual report, undergraduate participants may not have been familiar with the official NSF project 
title used in the survey item. For this reason, Item B2 included a “not sure” option: “Was this travel abroad while an 
undergraduate for the purpose of the [PROJECT TITLE] project?” The percent who answered “not sure” were as 
follows: 

• Research project: 1% 

• Internship: 8% 

• Attend a professional conference: 6% 

• Classes or educational activities abroad: 2%. 

^ N=191 had traveled abroad to help conduct an original research project. 

^ N=103 had traveled abroad to participate in an internship. 

N=83 had traveled abroad to attend a professional conference. 

^ N=1 13 had traveled abroad to take classes at an educational institution or participate in educational activities abroad. 

® N=209, Other U.S. -based undergraduates, missing=l; U.S. -based faculty or postdocs, missing=l. 

’ Average number of trips: N=208, missing=l. 

* Average duration of trips abroad: N=208, missing=l. 

* Longest trip abroad: N=208, missing=l. 

Among respondents who enrolled in courses or participated in educational activities abroad related to 
their PIRE project while traveling abroad, the majority took courses, seminars, or lectures in their 
primary major (78 percent) and studied a foreign language (57 percent; see Exhibit 4.17). Less than 
half of these respondents reported taking courses, seminars, or lectures outside their primary major, or 
studying the history, geography, and culture of the local country or region (43 percent each). 
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Exhibit 4.17: Types of Coursework Taken Outside the U.S. by Undergraduate PiRE Participants 
Who Participated in Coursework Abroad (N=60) 


Type of Coursework 

Percent 

Courses, seminars, lectures, or other education in my primary major or field of study 

78% 

Foreign language classes or education 

57% 

Courses, seminars, lectures, or other education in fields other than my primary major field of study 

43% 

Classes or other education about the history, geography, and culture of the country/region 

43% 

Other 

2% 


Source: Undergraduate Student Survey, Item BIO. 
Notes: N=60. 


Respondents with internships abroad related to PIRE typically worked in a college, university, or 
other institution of higher education (88 percent) and conducted research or research and development 
(99 percent). Exhibit 4.18 summarizes all settings and types of work these undergraduates reported. 


Exhibit 4.18: internship Setting and Type of Work invoived Among Undergraduate PiRE 
Participants with an internship Abroad (N=76) 


Internship Setting and Type of Work 

Percent 

College, university, or other institution of higher education 

88% 

Government agency or non-governmental organization 

12% 

Another setting 

7% 

Industry or business 

5% 

Medical or health care institution 

3% 

Elementary or secondary school 

0% 

Type of Work 

Research or research & development (R&D) 

99% 

Teaching, training, or mentoring others 

8% 

Providing help or services to others 

7% 

Policy-related work 

1% 

Another type of work 

1% 


Source: Undergraduate Student Survey, Item BII. 
Notes: N=76. 


4.2.2 Coiiaborations 

Undergraduate participants reported on collaborations during and subsequent to their participation in 
the project with individuals who were primarily based within and outside the U.S. Only 
undergraduates who traveled outside the U.S. for PIRE-related research were asked about their 
collaborations. 

Collaborations during the project with U.S. and non-U. S. project colleagues 

Among undergraduate respondents who traveled outside the U.S. for PIRE-related research, a greater 
proportion collaborated with foreign faculty, senior- or postdoctoral-level researchers, or graduate 
students than with foreign undergraduate students (85 percent compared to 39 percent; Exhibit 4.19). 


Exhibit 4.19: Undergraduate Coiiaboration During PiRE Project with individuais Based Outside 
the U.S. (N=172) 


Type of Individual Based Outside the U.S. 

Percent 

Faculty, senior- or postdoctoral-level researchers or graduate students based primarily at an 
institution outside the U.S. 

85% 

Undergraduate students based primarily at an institution outside the U.S. 

39% 


Source: Undergraduate Student Survey, Item Cl. 

Notes: N=172. Only undergraduates who traveled outside the U.S. for PIRE-related research were asked about their 
collaborations. 
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Collaborations subsequent to participation in the project 

A key outcome for the study is the extent to which PIRE undergraduate students continued 
collaborations with non-U.S. -based colleagues after their participation in the project had ended. In 
addition, the study collected descriptive data on the nature of these continued collaborations and any 
subsequent collaborations with foreign colleagues who were not part of the PIRE or comparison 
project teams. Only participants in expired projects (and participants who reported that their 
contributions to the project had ended prior to the survey) were asked to report on these outcomes. 

Less than half of undergraduate respondents who traveled outside the U.S. for PIRE-related research 
and collaborated with a foreign partner on research during their participation in the PIRE project went 
on to collaborate with a foreign partner after their participation in the project ended. A higher 
proportion of these respondents collaborated with a non-PIRE affiliated foreign partner (33 percent) 
than with a member of the project team (20 percent). 


Exhibit 4.20: Coiiaboration, After Participation in the Project, with a Non-U.S.-Based Partner 
(N=142) 


Type of Non-U.S.-Based Partner 

Percent 

Foreign project member affiliated with the PIRE project^ 

20% 

Foreign partner not affiliated with the PIRE project 

33% 


Source: Undergraduate Student Survey, Items C2a & C2d. 


Notes: 

* N=142. Non-U. S. -based project members were restricted to include only project members who “were based outside the 

U.S.” and who “still are” based outside the U.S., to the best of the respondent’s knowledge at the time of the survey. 
When the definition of non-U. S. -based colleague is expanded to include both (a) project members who were and still 
are based outside the U.S. and (b) project members who were based outside the U.S. but who “are now” based 
primarily within the U.S., the results change by less than three percentage points. 

^ N=142. Only undergraduates who traveled outside the U.S. for PIRE-related research were asked about their 

collaborations, and of those, only those who had participated in an expired PIRE award or those who reported that their 
participation in an active award had ended prior to the survey were asked about collaborations subsequent to their 
project participation. 

Among undergraduate respondents who traveled outside the U.S. for PIRE-related research and 
collaborated with a foreign member of the project team after their participation in the PIRE project 
ended, more than half reported that they discussed developments in their research field with a foreign 
partner (55 percent) and shared data or other resources with a foreign partner (52 percent). Just under 
half of respondents traveled abroad to collaborate with a foreign partner (48 percent) and conduct 
research with a foreign partner (42 percent). Approximately one -quarter of participants planned a 
research project with a foreign partner (27 percent) and about one-fifth met with a foreign partner at a 
conference outside the U.S. (18 percent). 
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Exhibit 4.21 : Percent of Undergraduates Reporting Seiected Types of Coiiaborative Activities 
with Non-U.S.-Based Coiieague from the Project Team, After Participation in the Project Had 
Conciuded (N=33) 


Type of Activity 

Percent 

Discussed developments in our research field(s) with a foreign partner 

55% 

Shared data or other resources with a foreign partner 

52% 

Traveled outside the U.S. to work with one or more foreign partners 

48% 

Conducted one or more research projects with a foreign partner 

42% 

Planned one or more research projects with a foreign partner 

27% 

Met with one or more foreign partners at conferences outside the U.S. 

18% 

A foreign partner has (or had) enrolled as a student at an institution in the U.S. 

9% 

Have (or have had) a job or position at an institution outside the U.S. 

6% 

A foreign partner has (or has had) a job at an institution in the U.S. 

6% 

Have enrolled as a student at an institution outside the U.S. 

3% 

A former U.S.-based partner has (or has had) a position at an institution outside the U.S. 

3% 

None of the above 

21% 


Sources: Undergraduate Student Survey, Item C2c. 

Note: N=33. Only undergraduates who traveled outside the U.S. for PIRE-related research were asked about their 
collaborations, and of those, only those who had participated in an expired PIRE award or those who reported that their 
participation in an active award had ended prior to the survey were asked about collaborations subsequent to their project 
participation. Those who reported a collaboration with a foreign member of the PIRE project after their participation in the 
project had ended were asked about the type of collaborative activity. 

4.2.3 Undergraduates’ Role in the Project 

Among undergraduate respondents who traveled outside the U.S. for PIRE-related research, most 
reported that during their participation in the PIRE project, they collected data (80 percent) and 
analyzed data (68 percent). Almost half of respondents reported that they prepared samples or 
specimens (48 percent), and approximately one-third reported that they wrote part of a manuscript or 
report on research (34 percent), conducted computer or mathematical simulations (32 percent), or 
designed research (30 percent). 


Exhibit 4.22: Percent of Undergraduates Reporting Various Roles in the PIRE Project (N=172) 


Role 

Percent 

Collecting data 

80% 

Analyzing data 

68% 

Preparing samples or specimens 

48% 

Writing part of a manuscript or report on the research 

34% 

Conducting computer or mathematical simulations 

32% 

Designing the research 

30% 

Synthesizing or fabricating materials 

20% 

Editing drafts of manuscript written by others 

20% 

Manufacturing or building a prototype or model 

12% 

Testing a prototype 

11% 

Other 

2% 

None of the above 

7% 


Source: Undergraduate Student Survey, Item D3. 
Notes: N=172. 


4.2.4 Undergraduates’ Contributions to PIRE Publications 

Among undergraduate participants in the PIRE project who traveled outside the U.S. for PIRE-related 
research, few reported that they had been, or would he, listed as a co-author on a PIRE-related 
publication (14 and 12 percent, respectively). Among undergraduate co-authors on a PIRE 
publications, 75 percent reported one such publication, with one-quarter reporting six or more; the 
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average number of publications was one. For tbe undergraduates anticipating co-authorsbip, 80 
percent reported one publication in preparation, under review or in press and 20 percent reported two 
such publications. On average, 90 percent of these publications and 60 percent of the publications not 
yet published included a foreign co-author. 


Exhibit 4.23: Percent of Undergraduate Students Reporting that They Have Been Or Wiii Be 
Listed as a Co-Author on a Paper Resuiting from the Project (N=172) 


Actual or Anticipated Co-authorship 

Percent 

I have been listed as a co-author on a publication 

14% 

I will be listed as a co-author on a paper that is in preparation, under review for possible publication or in 
press (accepted for publication but has not yet appeared) 

12% 


Source: Undergraduate Student Survey, Items D1 and D2a. 
Notes: N=172. 


Among undergraduate participants in the PIRE project who traveled outside the U.S. for PIRE-related 
research, more than half (58 percent) reported contributing to at least one conference abstract, poster 
or presentation. Exhibit 4.24 shows the distribution of the number of undergraduates contributing as a 
co-author on a conference abstract, poster or presentation (the average undergraduate contributed to 
one such product). On average, 60 percent of these conference -related materials included a foreign 
author. Less than 2 percent of PIRE undergraduates who traveled abroad for PIRE-related research 
reported contributing to a patent, but 15 percent of these undergraduates reported contributing to 
some other product that had been disseminated to a scholarly community or the public (e.g., 
educational materials, software, dataset or database). 

Exhibit 4.24: Distribution of Number of Co-Authored Conference Abstracts Reported by PiRE 
Undergraduates Who Traveied Abroad to Conduct Research (N=172) 
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Source: Undergraduate Student Survey, Item D2b. 

Notes: N=172, missing=12. Only under undergraduates who had traveled abroad for the PIRE project were asked about any 
co-authored publications, conference papers, patents or other products. 

4.2.5 Mentoring 

Among undergraduate respondents who traveled outside the U.S. for PIRE-related research, the 
majority of students reported that a U.S. -based faculty member acted as their primary mentor (70 
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percent). Another 19 percent reported that a non-U.S. -based faculty memher or senior-level 
researcher served in this role, and 10 percent reported having a graduate student mentor. 


Exhibit 4.25: Undergraduate Students’ Primary Mentor for Research Reiated to PiRE Project 
(N=172) 


Primary Mentor 

N 

% 

A U.S.-based faculty member (or other senior-level researcher, such as a postdoc) 

121 

70% 

A faculty member or senior-level researcher based primarily outside the U.S. 

32 

19% 

A graduate student based primarily outside the U.S. 

9 

5% 

A U.S.-based graduate student 

8 

5% 

Other 

1 

1% 

I did not have a mentor for this research 

1 

1% 


Source: Undergraduate Student Survey, Item B15. 
Notes: N=172. 


Respondents reported high levels of satisfaction with their PIRE mentors. The majority reported that 
their mentor helped them acquire research methods and techniques, and demonstrated a sincere 
interest in their learning (92 percent each). 


Exhibit 4.26: Undergraduate Students’ Opinions on Primary Mentor Whiie Working on PiRE 
Project (N=171) 



Stron 

Disagr 

Disag 

giy 

ee or 
ree 

Strongl; 
or A 

I Agree 
gree 

Mentor’s Contribution to Student 

N 

% 

N 

% 

Mentor helped me acquire the methods or techniques needed for the 
research 

14 

8% 

157 

92% 

Mentor showed sincere interest in my learning 

14 

8% 

157 

92% 


Source: Undergraduate Student Survey, Item BI6. 
Notes: N=171. 


4.2.6 Undergraduates Who Enroiied in, or Pianned to Enroii in, Graduate Schooi 

Among undergraduate respondents who earned their bachelor’s degrees prior to the survey date, 
approximately half reported that they were currently enrolled in a graduate degree program (54 
percent). The majority of these students reported that they expect to earn a doctoral degree (61 
percent), while approximately one -third expect to earn a master’s degree (30 percent). Ten percent of 
respondents expect to earn a professional degree (8 percent) or a post-baccalaureate certificate (2 
percent). 


** Less than 5 percent were in graduate school outside the U.S. 
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Exhibit 4.27: Percent Enroiied in and Degree Expected to Earn in Graduate Degree Program 


Graduate Degree Expectations 

N 

% 

Currently enrolled In graduate degree program^ 

132 

54% 

Degree expected^: 

Post-baccalaureate certificate (e.g., teaching certificate) 

2 

2% 

Master's degree (Includes MBA, MEd, etc.) 

39 

30% 

Doctoral degree (PhD, EdD, etc.) 

81 

61% 

Professional degree (only Includes: chiropractic, dentistry, law, 
medicine, optometry, osteopathic medicine, pharmacy, podiatry, 
ministry or divinity, or veterinary medicine) 

10 

8% 


Source: Undergraduate Student Survey, Items E3 & E4a. 


Notes: 

’ Only participants who had completed their bachelor’s degree (N=246) were asked about graduate school enrollment. 

Currently enrolled: N=244, missing=2. 

^ Degree expected: N= 132. 

The majority of respondents enrolled in a graduate program reported that their field of study is in the 
sciences, mathematics, or engineering fields (77 percent). Within this group, engineering was the 
most commonly reported suhfield (25 percent), followed hy biological sciences (13 percent), physics 
(12 percent), and earth, atmospheric, and ocean sciences (11 percent). 


Exhibit 4.28: Fieid of Degree for Graduate Program Among Former PiRE Undergraduates 
Currentiy Enroiied in Graduate Schooi (N=132) 


Field of Expected Graduate Degree 

N 

% 

Sciences, mathematics, or engineering 

102 

77% 

Medical or health sciences 

12 

9% 

Social sciences 

10 

8% 

Other fields 

6 

5% 

Humanitles/arts 

1 

1% 

Business, management 

1 

1% 


Source: Undergraduate Student Survey, Item E4c. 

Notes: N=132. See the Undergraduate Student Survey for the list of subfields associated with the major disciplines above. 


Among respondents who had earned their bachelor’s degree but were not enrolled in a graduate 
program as of the survey date, the majority reported that they intend to pursue a master’s or doctoral 
degree in a STEM field (43 percent). Fourteen percent of respondents reported that they do not intend 
to pursue any additional education and 12 percent intend to pursue a medical degree. 


Exhibit 4.29: Educationai Pians Beyond Bacheior’s Degree (N=76) 


Educational Plans 

N 

% 

A master's or doctoral degree In engineering, computer science, mathematics or the natural 
(biological or physical) sciences 

33 

43% 

A medical degree (MD) 

9 

12% 

A master's or doctoral degree In the social sciences 

6 

8% 

A business degree (MBA) 

6 

8% 

A master's or doctoral degree In a field other than engineering, computer science, math, the 
natural or social sciences 

4 

5% 

A joint medical and doctoral degree (MD/PhD) 

1 

1% 

Another health-related or professional counseling degree (e.g., nursing, Psy.D., social work) 

1 

1% 

A teaching certificate or master's degree related to a field In which 1 teach/wlll teach 

1 

1% 

None of the above: a different graduate degree or certificate 

4 

5% 

1 do not Intend to pursue a graduate degree or certificate 

11 

14% 


Source: Undergraduate Student Survey, Item E5. 
Notes: N=76. 
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4.2.7 Reported Benefits and Challenges of PIRE 

Among undergraduate students who traveled outside the U.S. for PIRE-related research, 82 percent of 
respondents agreed or strongly agreed that different cultural perspectives improved the quality of the 
research. The majority of students reported that neither administrative procedures nor language 
differences hindered the progress of their project’s research (80 percent and 82 percent, respectively). 


Exhibit 4.30: Reported Benefits/Challenges While Participating in PIRE Project (N=172) 



Strongly 
Agree or 
Agree 

Stro 

Disag 

Disa 

ngly 
ree or 
gree 

Not 

Applicable 

Benefit/Challenge 

N 

% 

N 

% 

N 

% 

Different cultural perspectives improved the quality of the research 

140 

82% 

13 

8% 

18 

11% 

Regulatory, bureaucratic or administrative procedures in a foreign 
country hindered the progress of the research 

30 

18% 

136 

80% 

5 

3% 

Language differences hindered the progress of the research 

27 

16% 

140 

82% 

4 

2% 


Source: Undergraduate Student Survey, Item B1 7. 
Notes: N=172. 


Among those who traveled abroad, 70 percent reported facing no challenges (Exhibit 4.31). The 
challenges most commonly reported by undergraduates were not enough opportunity to contribute to 
research and not enough supervision or mentoring (10 percent each). 


Exhibit 4.31: Reported Challenges Encountered While Participating in PIRE Project (N=172) 


Challenges 

N 

% 

None 

121 

70% 

Not enough opportunity to contribute to the research 

17 

10% 

Not enough supervision or mentoring 

17 

10% 

Not enough information or guidance to prepare for the foreign component of the research 

11 

6% 

Other— specify 

10 

6% 

Unexpected financial burdens 

9 

5% 

Delays pursuing/completing my bachelor's degree 

8 

5% 

Bias from foreign members of the research team based on my nationality, gender, race, ethnicity, 
religion, disability 

6 

3% 

My contributions to the research were not always appreciated or respected 

3 

2% 

Too much supervision 

1 

1% 


Source: Undergraduate Student Survey, Item BIS. 
Notes: N=172. 


Among undergraduate respondents who traveled outside the U.S. for PIRE-related research, the 
majority reported that their participation in the project yielded a range personal benefits (Exhibit 
4.32). Most commonly reported among these was an understanding of how scientists work on real 
problems (81 percent), followed by tolerance for obstacles faced in the research process (71 percent), 
self-confidence in their field of study (69 percent), and clarification of a career path (67 percent). 
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Exhibit 4.32: Reported Benefits of PiRE Participation (N=172) 


Benefit of Participation 

Very Large to 
Large Benefit 

Moderate to 
Small Benefit 

No Benefit 

Not 

Applicable 

N 

% 

N 

% 

N 

% 

N 

% 

Understanding of how scientists work on 
reai probiems 

140 

81% 

31 

18% 

0 

0% 

1 

1% 

Toierance for obstacies faced in the 
research process 

122 

71% 

45 

26% 

2 

1% 

3 

2% 

Seif-confidence in my fieid of study 

118 

69% 

43 

25% 

9 

5% 

2 

1% 

Ciarification of a career path 

115 

67% 

53 

31% 

2 

1% 

2 

1% 


Source: Undergraduate Student Survey, Item D4. 
Notes: N=172. 


Ninety-two percent of PIRE undergraduate participants indicated that they would strongly 
recommend international research or education to their peers, with an additional 6 percent indicating 
they would recommend it. Only 3 percent indicated they would recommend it with reservations, and 
no undergraduates indicated they would not recommend it. 

In explaining why they would recommend participation in PIRE to others, undergraduates cited PIRE 
as an opportunity for: broadening of research/academic perspectives (62 responses); cultural and 
social exchange (45 responses); enhancing knowledge and skills (31 responses); professional growth 
and improves post-graduate opportunities (28 responses); expanding ability to conduct relevant/hands 
on research (23 responses); personal growth (20 responses); gaining a better understanding of 
research opportunities/field (18 responses); developing important connections or networks (14 
responses); learning from scientific experts (6 responses); and international travel (5 responses). 

When asked about the specific aspects of their participation in PIRE that had been most beneficial to 
them, undergraduate students who traveled abroad for the project described: experiences 
collaborating with international researchers (34 responses); development of new tools/skills (26 
responses); enhanced network (26 responses); role in clarifying/aiding career goals (27 responses); 
opportunity to live/travel abroad (23 responses); cultural and social exchange (22 responses); 
improved understanding of research experience/process (21 responses); exposure to new 
research/academic perspectives (19 responses); language acquisition (13 responses); personal growth 
(10 responses); experience presenting findings (8 responses); and relationship with mentor (5 
responses). 

Specific aspects of their PIRE participation that they cited as most challenging included: specific 
research issues (28 responses); limited knowledge or experience needed for project (22 responses); 
communication or cultural barriers (34 responses); issues with collaborative partnerships (16 
responses); issues related to travel (9 responses); lack of guidance or support (9 responses) lack of 
time (8 responses); and issues related to resources or equipment (5 responses). 


4.3 Foreign Senior Investigators in PIRE 

Surveys of PIRE senior investigators based in foreign institutions provided data on the perspectives of 
the experiences of participants in partner countries. 

4.3.1 Travel for the PIRE Project 

Less than half of foreign investigators traveled internationally during their participation in the PIRE 
project (48 percent; Exhibit 4.33). Among those who traveled abroad, the majority traveled to the 
U.S. (78 percent). On average, foreign investigators took 3 to 4 international trips during their 
participation in the project. 
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Exhibit 4.33: Percentage of Foreign investigators Who Traveied Abroad During the PiRE 
Project 


Traveled Abroad 

N 

Percent 

Traveled internationally during project^ 

58 

48% 

Among those who traveled, traveled to U.S.^ 

45 

78% 

Number of International Trips 

Mean 

SD 

Among those who traveled, average number of international trips^ 

3.4 

3.0 


Source: Foreign Senior Investigator Survey, Items C8, C9 and ClOa. 
Notes: 

' N=120, missing=2. 


^ N=58. 

^ N=57, missing=3. 

In addition to their own travel, foreign investigators were also asked to report whether other project 
staff at their institution traveled internationally and whether any of that travel was to the U.S. Forty- 
eight percent of respondents reported that one or more members of their institution traveled abroad 
for PIRE, and 84 percent reported that team members who traveled abroad visited the United States. 
The most commonly reported group of individuals to travel abroad was senior researchers (72 
percent), followed by graduate students (63 percent), postdoctoral researchers (37 percent), and 
undergraduate students (10 percent). Similar proportions of foreign investigators reported that senior 
researchers and graduate students traveled to the U.S. (69 percent and 61 percent, respectively). In 
comparison, fewer respondents reported that postdoctoral researchers and undergraduate students 
visited the U.S. (42 percent and 14 percent, respectively). 


Exhibit 4.34: Types of individuais from Fi’s institution Who Traveied internationaiiy and Visited 
the U.S. for the PiRE Project 



Internationally 
for PIRE Project 

To the U.S. 
for PIRE Project 

Type of Individual Who Traveled Abroad 

N of Foreign 
investigators 

% 

N of Foreign 
investigators 

% 

Any member of FI's institution traveled outside home 
country^ 

49 

91% 

46 

84% 

FI reported that one or more undergraduate students 
traveled... 2 

5 

10% 

7 

14% 

FI reported one or more graduate students traveled... ^ 

31 

63% 

31 

61% 

FI reported one or more postdoctoral researchers 
traveled...'* 

19 

37% 

21 

42% 

FI reported one or more senior researchers traveled... ^ 

39 

72% 

38 

69% 


Source: Foreign Senior Investigator Survey Item CI2. 


Notes: N indicates the number of foreign senior investigators who reported that a certain member from their institution 
traveled internationally and/or to the U.S. for the project. 

* Traveled internationally: N=54, missing=5; visited the U.S.: N=55, missing=4. 

^ Traveled internationally: N=50, missing=9; visited the U.S.: N=49, missing=10. 

^ Traveled internationally: N=49, missing=10; visited the U.S.: N=51, missing=8. 

^ Traveled internationally: N=51, missing=8; visited the U.S.: N=50, missing=9. 

Traveled internationally: N=54, missing=5; visited the U.S.: N=55, missing=4. 

4.3.2 Collaborations 

Foreign investigators reported on collaborations before, during, and after to their participation in the 
project with individuals who were primarily based inside the U.S. 

Collaborations during the project with U.S. project colleagues 

The majority of respondents reported that they collaborated with PIRE team members based within 
the U.S. The highest proportion of foreign investigators reported working with senior-level 
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researchers (94 percent), followed by graduate or undergraduate students (81 percent) and 
postdoctoral researchers (69 percent). 


Exhibit 4.35: Percent of Fis Who Coiiaborated During the Project with individuais Based within 
the U.S. 


Type of U.S.-Based Partner 

N 

% 

Senior-level researchers based primarily in the U.S.'' 

114 

94% 

Postdoctoral researchers based primarily in the U.S.^ 

82 

69% 

Graduate or undergraduate students based primarily in the U.S.^ 

96 

81% 


Source: Foreign Senior Investigator Survey, Item Bla. 


Notes: 

’ Senior-level researchers: N=121, missing=l. 

^ Postdoctoral researchers: N=1 19, mrssing=3. 

^ Graduate or undergraduate students: N=1 19, missing=3. 

Percent of FIs who had a collaboration before the start of the PI RE project 

For most foreign investigators, PIRE was not their first research collaboration with a U.S. -based 
partner. Eighty-two percent of respondents reported working with a U.S. -based individual on research 
prior to their participation in the PIRE project. Among these respondents, 69 percent reported that 
they collaborated with a U.S. -based partner with whom they would eventually work on the PIRE 
project. Ninety-four percent reported that their pre-PIRE collaboration had existed for one year or 
longer. 


Exhibit 4.36: internationai Research Coiiaborations Prior to Participation in PiRE 


Collaborated Prior to PIRE Project with: 

N 

% 

U.S.-based partner(s)' 

94 

82% 

U.S.-based partner(s) who also participated in PIRE project 

65 

69% 

Of those collaborated prior to PIRE with U.S-based PIRE colleague:'’ 

Duration of prior collaboration was 1 year or more 

61 

94% 

Duration of prior collaboration was 1 to 12 months 

4 

6% 


Source: Foreign Senior Investigator Survey, Items DI, D2, D3. 
Notes: 


N=1 14, missing=8. 

^ N=94. 

^ N=65. 

Collaborations subsequent to participation in the project 

A key outcome for the study is the extent to which PIRE foreign investigators continued 
collaborations with U.S. -based colleagues after their participation in the project had ended. In 
addition, the study collected descriptive data on the nature of these continued collaborations and any 
subsequent collaborations with U.S-based colleagues who were not part of the PIRE or project teams. 
Only participants in expired projects (and participants who reported that their contributions to the 
project had ended prior to the survey reference date) were asked to report on these outcomes. 

Foreign investigators continued to collaborate with U.S. -based researchers after their participation in 
the PIRE project. Eighty-four percent of respondents reported collaborating with a U.S. -based 
member of the PIRE team who was and still is based in the U.S., and 32 percent reported working 
with a U.S. -based member of the PIRE team who was based in the U.S. but is currently based outside 
the U.S. (Exhibit 4.37). Additionally, 77 percent of foreign investigators reported collaborating with a 
non-PIRE U.S. -based partner after the end of their project participation. 
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Exhibit 4.37: Coiiaboration, After Participation in the Project, with a U.S.-Based Partner 


Type of U.S.-Based Partner 

N 

% 

One or more project members who were— and are still— based primarily in the U.S.'' 

51 

84% 

One or more project members who were based primarily in the U.S. but who are now based 
primarily outside the U.S. ^ 

18 

32% 

With a U.S. partner not affiliated with PIRE project' 

47 

77% 


Source: Foreign Investigator Survey, Item Bib & Ble. 
Notes: 


' N=61. 

^ N=57, missing=4. 

Types of Collaborative Activity After Participation in PIRE 

Foreign investigators reported engaging in a range of different activities with their U.S. -based 
partners after their participation in the PIRE project (Exhibit 4.38). Most common among them was 
discussing developments in the research field (83 percent), followed by sharing data or other 
resources (76 percent), meeting with one or more partners at a U.S. conference (67 percent), working 
with a U.S. -based partner outside the U.S. (63 percent), and planning and conducting research 
project(s) (57 percent each). Eess than half of respondents reported traveling to the U.S. (46 percent), 
and less than one -tenth reported having a position in the U.S. (9 percent) or working with a U.S.- 
based partner with a position outside the U.S. (7 percent). 


Exhibit 4.38: Percent Reporting Seiected Types of Coiiaborative Activities with U.S.-Based 
Coiieague from the Project Team, After Participation in the Project Had Conciuded (N=54) 


Type of Activity 

N 

% 

Discussed developments in our research field(s) with a U.S.-based partner 

45 

83% 

Shared data or other resources with a U.S.-based partner 

41 

76% 

Met with one or more U.S.-based partners at conferences in the U.S. 

36 

67% 

A U.S.-based partner has worked with me at location(s) outside the U.S. 

34 

63% 

Planned one or more research projects with a U.S.-based partner 

31 

57% 

Conducted one or more research projects with a U.S.-based partner 

31 

57% 

Traveled to the U.S. to work with one or more U.S.-based partners 

25 

46% 

Have (or have had) a position at an institution within the U.S. 

5 

9% 

A U.S.-based partner has (or has had) a position at an institution outside the U.S. 

4 

7% 


Source: Foreign Investigator Survey Item Bid. 
Notes: N=54. 


4.3.3 Foreign investigators’ Funding for Participation in the PiRE project 

PIRE award funds are restricted to U.S. personnel; as a result, foreign investigators needed their own 
resources to participate in the PIRE project. Beginning with the competition for the 2012 cohort of 
PIRE awards, NSE began working with counterpart funding agencies in a number of countries to 
enhance opportunities for Els to obtain funding (the U.S. Agency for International Development and 
the U.S. Environmental Protection Agency were among these counterpart agencies). Although NSF’s 
outreach to these agencies to support PIRE awards began in 2012, investigators were likely aware of 
these counterpart agencies as a potential funding source before then. Thus, we asked all FIs 
(regardless of PIRE cohort) whether they obtained funding from any of these counterpart agencies or 
from another source. 

Nearly one -quarter of Els reported that they received funding from none of the sources listed 
(including any other source not listed which they could specify). Among the most commonly reported 
source of funding to support PIRE activities was foreign investigators’ institution or employer (44 
percent), followed by a government or institution in another country (38 percent). Eess than 20 
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percent of foreign investigators reported that they received funding from each of several other 
potential sources listed in Exhibit 4.39, helow. 


Exhibit 4.39: Sources and Uses of Funding Received for Project Participation 


Funding Sources and Uses of Funding 

N 

% 

Funding Sources^ 

My institution or empioyer 

53 

44% 

Another government or nationai institution not among those iisted in the survey^ 

46 

38% 

An internationai institution (not among those iisted) 

13 

11% 

U.S. Agency for Internationai Deveiopment (USAiD) 

4 

3% 

Private industry (domestic, foreign, or muitinationai company) 

3 

2% 

A foundation or phiianthropic organization (e.g.. Gates Foundation) 

2 

2% 

U.K. Engineering and Physicai Sciences Research Councii (EPSRC) 

1 

1% 

Ministry of Education and Science (MES) of the Russian Federation (RF) 

1 

1% 

U.S. Environmentai Protection Agency (US EPA) 

1 

1% 

Other 

19 

16% 

U.K. Economic and Sociai Research Councii (ESRC) 

0 

0% 

Japan Science and Technoiogy Agency (JST) 

0 

0% 

Inter-American institute for Giobai Change Research (lAI) 

0 

0% 

None of the above 

29 

24% 

Funding Uses^ 

To support my work on the project 

67 

55% 

To pay for equipment, suppiies, resources or faciiities, or for access to such resources 

51 

42% 

To support one or more graduate students working on the project 

46 

38% 

For travei outside my country that was necessary to conduct the research 

44 

36% 

For travei to professionai conferences (either outside my country or within my country) 

41 

34% 

For expenses reiated to hosting internationai guests (e.g., housing, meais, etc.) 

39 

32% 

For travei within my country that was necessary to conduct the research 

38 

31% 

None 

28 

23% 

To support one or more postdoctorai researchers working on the project 

26 

21% 

To support educationai activities such as courses, iecture series, seminars 

23 

19% 


Sources: Foreign Senior Investigator Survey, Items Cl and C2. 


Notes: 

’ Funding sources: N=121, missing=l. 

^ The survey listed counterpart agencies in other countries with whom NSF works to coordinate joint international 
research and educational efforts. The survey included an option for FIs to indicate “another government or national 
institution” that was not among the counterpart agencies listed in the survey but from which they had received funding. 
^ Funding uses: N=121, missing=l. 

The most commonly reported use of funding that FIs did obtain was to support foreign investigators’ 
own work on the project (55 percent), followed by purchasing equipment, supplies, resources or 
facilities (42 percent). Approximately one-third of respondents reported that they used these funds to 
support graduate students (38 percent), to travel abroad for research (36 percent) or to a conference 
(34 percent), to house international guests (32 percent), and to travel within their country to conduct 
research (31 percent). Roughly one-fifth of foreign investigators used these funds to support 
postdoctoral researchers (21 percent) or educational activities (19 percent). 

Almost half of foreign investigators reported receiving less than $50,000 to support PIRE activities 
(Exhibit 4.40), while another 20 percent reported receiving more than $100,000 and 7 percent 
reported receiving a figure between those two amounts. 
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Exhibit 4.40: Totai Estimated Amount of Funding Received to Support Research or 
Educationai Activities for the PiRE Project (in USD) (N=59) 


Amount in USD 

N 

% 

Less than 50,000 USD 

29 

49% 

50,001 to 100,000 USD 

4 

7% 

More than 100,000 USD 

12 

20% 


Source: Foreign Senior Investigator Survey, Items C3a & C3b. 

Notes: N=59, missing=l. Fourteen (24 percent) FIs who answered in USD did not know the total amount of funding 
received. Respondents were asked if they could report the total amount of funding received in terms of U.S. dollars. Of 122 
FIs respondents who reported that they had obtained funding (Item Cl) to support their PIRE participation, 60 provided an 
amount in U.S. dollars. 

4.3.4 Foreign Senior investigators’ Roie in the Project 

The majority of foreign investigators report that they were involved in the planning, designing or 
conceptualizing of the PIRE project activities (78 percent). Among these respondents, 88 percent met 
in person with project team members based outside their own country. Among the 22 percent of 
foreign investigators not involved in the initial planning stages, the majority reported that their 
exclusion resulted in either no or only minor difficulties with the project (92 percent). 


Exhibit 4.41 : Foreign investigators’ Participation in Project Pianning 


Role in Project Planning 

N 

% 

Involved in planning, designing or conceptualizing the research or educational 
activities for the project^ 

93 

78% 

During planning of the project, met in person with project colleagues based outside 
your own country (for those who were involved in planning)^ 

83 

88% 


Source: Foreign Senior Investigator Survey, Items C4-C6. 

Notes: 

1 N=1 19, missing=3. Respondents who answered “yes” [involved in planning, designing or conceptualizing the research 


or educational activities for the project] (93) or who did not answer the question (3) were directed to Item C6 [During 
planning of the project, met in person with project colleagues based outside their own country]. 

2 N=94, missing=2. 

3 N=26. 26 (22%) respondents were not involved in planning, designing or conceptualizing the research or educational 
activities for the project. 

Foreign investigators were asked to report which PIRE project team member(s) had primary 
responsibility for key aspects of project research — both U.S. -based and non-U.S. -based partners, only 
U.S. -based partners, or only non-U.S. -based partners. For all but two research activities listed in 
Exhibit 4.42 below, the greatest proportion of respondents reported that responsibility was shared 
equally by U.S.- and non-U.S. -based team members. Foreign investigators reported that U.S. -based 
partners had primary responsibility for developing ideas, hypotheses, or vision for the project (53 
percent) and coordinating the multi-national team of researchers (56 percent). 
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Exhibit 4.42: Percent of Fis Reporting Who Had Primary Responsibiiity for Key Project-Reiated 
Research Activities (N=113) 


Project-Related Research 
Activity 

U.S.-Based and 
Non-U.S.-Based 
Partners 
Shared 

Responsibility 
About Equally 

U.S.-Based 
Partners Had 
Primary 
Responsibility 

Non-U.S.-based 
Partners Had 
Primary 
Responsibility 

Not 

Applicable 
(this activity 
was not part 
of the 
research) 

Don't 

Know 

Interpreting results 

73% 

13% 

5% 

5% 

4% 

Disseminating results through 
publications and 
presentations 

71% 

19% 

5% 

1% 

4% 

Analyzing data 

65% 

15% 

9% 

7% 

4% 

Collecting data 

56% 

17% 

12% 

9% 

6% 

Developing instrumentation, 
software, equipment, or data 
collection approaches 

43% 

26% 

12% 

8% 

11% 

Developing the ideas, 
hypotheses, or vision for the 
research project 

42% 

53% 

3% 

0% 

3% 

Preparing samples or 
specimens 

38% 

18% 

16% 

22% 

7% 

Coordinating the multi- 
national team of researchers 

36% 

56% 

4% 

2% 

3% 

Synthesizing or fabricating 
materials 

23% 

15% 

11% 

34% 

18% 


Source: Foreign Senior Investigator Survey, Item CIS. 
Note: N=113, missing=8. 


4.3.5 Reported Benefits and Chaiienges of PiRE 

Foreign investigators were asked to report on the benefits and challenges experienced during their 
participation in the PIRE project. Almost all respondents agreed or strongly agreed that their research 
benefitted from access to the expertise of U.S. -based researchers (96 percent), and large proportions 
agreed or strongly agreed that their research benefitted from access to data acquired in foreign 
locations (81 percent), different cultural perspectives (76 percent), and access to equipment or 
facilities in foreign locations (71 percent). The majority of foreign investigators reported that neither 
administrative procedures nor language differences, hindered the progress of their project’s research 
(75 percent and 86 percent, respectively). 
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Exhibit 4.43: Benefits and Chaiienges of PiRE Project on Foreign Senior investigator's 
Research 


Benefit/Challenge 

Strongly 
or Ac 

Agree 

iree 

Strongly 
Disagree or 
Disagree 

N 

% 

N 

% 

My research benefited from access to expertise of U.S. researchers in my field 

109 

96% 

4 

4% 

My research benefited from access to data acquired in foreign locations 

92 

81% 

21 

19% 

Different cultural perspectives improved the quality of the research 

85 

76% 

27 

24% 

My research benefited from access to equipment or facilities in foreign locations 

79 

71% 

32 

29% 

My research benefited from access to place-based phenomena that occur 
outside my country (e.g., biological, geological, cultural phenomena) 

53 

47% 

59 

53% 

Regulatory, bureaucratic or administrative procedures for U.S. personnel [have 
hindered/hindered] the progress of the research 

28 

25% 

84 

75% 

Language differences hindered the progress of the research 

16 

14% 

96 

86% 


Source: Foreign Senior Investigator Survey, Item El. 
Note: Ns range from 1 11-113, missing range from 9-11. 


Similarly, when asked about a range of other potential challenges to participation in the PIRE project, 
77 percent reported that they did not experience any challenges (Exhibit 4.44). The most commonly 
reported specific challenge faced by foreign investigators was that expected funding was not 
obtained, was reduced, or was withdrawn (10 percent). 


Exhibit 4.44: Chaiienges Foreign Senior investigators’ Experienced During Project (N=114) 


Challenges 

N 

% 

None 

88 

77 

Funding expected was not obtained, was reduced or withdrawn 

11 

10 

Inadequate access to data, specimens, facilities, equipment, or other resources 

7 

6 

Delays in getting publications written due to the cross-cultural nature of the team 

4 

4 

Conflicts over authorship or credit for contributions to the research 

2 

2 

Bias from U.S.-based researchers based on my nationality, gender, race, ethnicity, 
religion, disability, nationality, or sexual orientation 

1 

1 

Key U.S.-based researchers were not as available as 1 expected or did not contribute 
enough to the project 

1 

1 

Other 

12 

11 


Source: Foreign Senior Investigator Survey, Item E3. 
Note: N=114, missing=8. 


Foreign investigators were overwhelmingly positive when asked about their U.S. -based partners 
(Exhibit 4.45). All respondents reported that they either agreed or strongly agreed that senior partners 
were sufficiently knowledgeable and expert to fully participate in the research collaboration. Foreign 
investigators agreed or strongly agreed that their U.S. -based partners integrated well with members of 
their group (94 percent) and demonstrated interest in the scientific culture and organization and the 
history, culture, and customs of their country (87 percent and 82 percent, respectively). 
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Exhibit 4.45: Perception of U.S. Coiieagues By Foreign Senior investigators (N=113) 


Perception of U.S. Colleagues 

Strongly Agree or 
Agree 

Strongly Disagree 
or Disagree 

Not 

Applicable 

N 

% 

N 

% 

N 

% 

Senior U.S.-based partners had sufficient knowiedge 
and expertise to be a fuii participant in a research 
coiiaboration 

113 

100% 

0 

0% 

0 

0% 

U.S.-based partners integrated weii with members of my 
group 

106 

94% 

4 

4% 

3 

3% 

U.S.-based partners showed interest in scientific cuiture 
and organization, and how it is organized and funded in 
my country 

98 

87% 

10 

9% 

5 

4% 

U.S.-based partners showed interest in iearning about 
the history, cuiture, and customs of my country 

93 

82% 

12 

11% 

8 

7% 


Source: Foreign Senior Investigator Survey, Item E2. 
Note: Note: N=113, missing=9. 
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5. Institutional Policies and Practices for International Research 
and Education 


From each of the 59 PIRE sponsoring institutions (i.e., the institution of the lead PI), we sought an 
administrator who could speak to the institutional policies and practices that may have influenced the 
operation of research and educational activities associated with their institution’s PIRE award. This 
chapter examines these institutional policies and practices and the influence that these policies and 
practices might have had on the PIRE project — and how PIRE might have affected any institutional 
changes. Specifically, the findings address the following research question: 

• RQ 5. How do international affairs representatives at PIRE lead Pis’ institutions perceive the 
effects of PIRE on their institutions’ policies and practices for supporting international research 
and educational collahorations? 

Surveys of institutional administrators provide insight into how PIRE projects operate within U.S. 
institutions of higher education (IHEs) and the role these projects played within IHEs. Key findings 
include: 

• Institutional representatives reported that their institutions support international research and 
education initiatives through a variety of mechanisms, and the majority indicated that policies and 
practices mostly facilitated graduate and undergraduate participation in international 
opportunities. 

• Institutional representatives found that PIRE aligned well with existing institutional goals for 
internationalization. 

• A greater number of representatives descrihed institutional factors that support undergraduate and 
graduate student involvement in education and research activities abroad than the number that 
described barriers to such involvement. However, more representatives mentioned barriers to 
undergraduate than graduate student participation in international opportunities. 

• The majority of administrators responded that PIRE had not resulted in any changes in policies or 
practices related to undergraduate or graduate education in STEM fields (32 respondents). 

Among the 14 responses that credited PIRE with effecting change, many indicated that faculty, 
students and administrators had increased awareness of the value of international research 
experiences for students. 

5.1 PIRE Institutions’ Mechanisms of Support for International Research 
and Education 

Administrators were asked whether their institution had supported international research and 
education initiatives through the specific mechanisms presented in Exhibit 5.1. The large majority of 
administrators responded that their institutions: assisted students with arrangements for 
research/education abroad (42 respondents); coordinated with partner institutions outside the U.S. 
(41); assisted faculty with international arrangements (40); had an existing international exchange 
program (40); provided assistance developing international components of proposals (37); and 
provided seed money to faculty planning international collaboration (33). 
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Exhibit 5.1 : Ways in which the PiRE Sponsoring institution Faciiitated internationai Research 
or Education initiatives 



Institutional Administrator 


Survey 


N 

% 

Assisted students with arrangements for research/education abroad 

42 

93% 

Heiped participants in internationai activities coordinate with a partner institution in 
one or more countries outside the U.S. 

41 

91% 

Assisted facuity with arrangements for internationai research/education abroad 

40 

89% 

Have existing exchange program with one or more countries outside the U.S. 

40 

89% 

Provided facuity writing grant proposais with assistance deveioping the internationai 
components of a proposai 

37 

82% 

Provided facuity pianning internationai research/education coiiaborations with seed 
money 

33 

73% 

Have a campus in a country outside the U.S. 

11 

24% 

Other 

5 

11% 


Source: Institutional Administrator Survey, Item Q6. 
Note: N=46. 


Administrators also described how PIRE fit within a larger institutional context. For example, 
administrators frequently cited the PIRE award as being aligned with the institution’s goal of 
expanding its global reach (15 responses) and/or with efforts to build international partnerships (13). 
Other respondents reported that the PIRE project fit the institution’s goal of providing students with 
new experiences and opportunities (9) and others that PIRE supported their institution’s goal of 
producing a strong record of research and publications (6). 

The PIRE program has complete synergy with our institution's strategy of expanding our global 
reach through sending more students abroad, hosting more international students and scholars 
and building long-lasting strategic partnerships with select institutions around the world. It 
convinces us that our philosophy of not developing offshore campuses is the right one for us. 
(Administrator at a 2005 PIRE awardee) 

One mandate for our institution is to expand its global reach, and in addition, international 
experiences are considered an asset in students' educations. The PIRE stands as an example of 
how interdisciplinary research and education at both the undergraduate and graduate levels can 
all come together. (Administrator at a 2010 PIRE awardee) 

Our institution strives for excellence and its goal is to become a "world's ahead" university. 
Being aware of other top international universities is a key to achieve this goal. PIRE has helped 
our institution to partner with some of the best academic research institutions in the world. We 
have learned a great deal from their success and we believe that we have made a great 
impression on them by sending some of our top student and faculty researchers to their 
institutions. (Administrator at a 2007 PIRE awardee) 

The PIRE award advanced the university's goals of advancing undergraduate research and 
research abroad. It provided a concrete model of an undergraduate research abroad program. 
(Administrator at a 2007 PIRE awardee) 

PIRE broadened the scope of joint international research and academic collaboration for our 
faculty and provided excellent opportunities for our students to conduct cutting-edge research in 
some of the best laboratories/institutions abroad. (Administrator at a 2012 PIRE awardee) 
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5.2 Institutional Factors Influencing Student Participation in International 
Research and Education 

Responses provided insight into institutional factors that facilitated participation of students in 
research or education outside the U.S. (Exhibit 5.2). The majority of institutional administrators 
perceived institutional policies and practices as “mostly facilitating” participation of STEM 
undergraduate and graduate students in international research or education activities (26 and 28 
respondents, respectively). More than one-quarter thought that policies and practices had mixed 
effects for undergraduates, with some policies facilitating and others hindering participation (12 and 7 
respondents, respectively), whereas just 15 percent (7 respondents) thought that there were mixed 
effects for graduate students. Small proportions of respondents reported that their institutional policies 
and practices mostly hindered or had no effect on participation hy undergraduate or graduate students. 


Exhibit 5.2: institutionai Poiicies or Practices that Faciiitate or Hinder Undergraduate or 
Graduate Student Participation in Research or Education Activities Outside the U.S. 



Undergraduate 

Students 

Graduate Students 

N 

% 

N 

% 

Policies or practices mostly facilitate participation 

26 

57% 

28 

61% 

Both: Some policies or practices facilitate participation and 
other policies or practices hinder participation 

12 

26% 

7 

15% 

Policies or practices mostly hinder participation 

4 

9% 

4 

9% 

Neither: Institutional policies or practices neither facilitate nor 
hinder participation 

3 

7% 

6 

13% 

Not applicable 

1 

2% 

1 

2% 


Source: Institutional Administrator Survey, Items Q2 & Q3. 
Note: N=46. 


5.2.1 Factors Faciiitating Undergraduate internationai Engagement 

Eorty-three respondents cited one or more factors that served to facilitate undergraduate engagement 
in international opportunities. These factors varied, hut multiple respondents mentioned general 
institutional support for international experiences for undergraduate students (13 responses). 

The University wholeheartedly embarked on a strategic path of expanding our global reach. This 
University-wide effort was largely faculty-driven and administratively-enabled. This encouraged 
faculty to look beyond the traditional study-abroad programs and creatively integrate 
international experiential learning into courses and research programs. This greatly facilitated 
global research opportunities for undergraduate students. (Administrator at a 2005 TIRE 
awardee) 

We are implementing a comprehensive internationalization program, which considers 
undergraduate overseas research participation highly valuable. (Administrator at a 2007 PIRE 
awardee) 

Administrators also cited their study ahroad/intemational programs office as a more specific example 
of the institution facilitating undergraduates’ participation in international research and education (13 
responses). 

Our Office of International Programs has several programs to facilitate UG visits to 
international universities. Some of these offer opportunities for research participation. Our 
Chancellor encourages international travel and engagement during every convocation. 
(Administrator at a 2005 PIRE awardee) 


Abt Associates 


Evaluation of NSF’s PIRE program: Volume 1 1 pg. 65 



Institutional Context 


[ Our institution] has an active Office of International Affairs with a strong Study Abroad 
component. Personnel and associated policies are designed to help students in multiple ways, 
including academically, financially, and procedurally. (Administrator at a 2007 PIPE awardee) 

The study abroad office works closely with the colleges / academic disciplines to facilitate 
undergraduate study/research abroad. The study abroad office helps students with the 
application process, visas, finances, etc. (Administrator at a 2010 PIPE awardee) 

Some administrators mentioned that their institution helped undergraduates find sourees of funding to 
support international experiences (6 responses) including scholarships, travel support, or financial 
subsidies for study abroad programs. 

Five administrators reported that their institution specifically promotes opportunities for international 
engagement to STEM students. 

My office designs and administers international exchanges and partnerships, and we actively 
promote participation of STEM students in our exchanges and specialized programs. We conduct 
orientations, produce manuals, have an advising infrastructure, and have committees dealing 
with health and safety abroad. (Administrator at a 2007 PIPE awardee) 

5.2.2 Factors Hindering Undergraduate internationai Engagement 

Twenty administrators listed policies or practices that restricted the opportunities for undergraduates 
to pursue international research or education. The factors cited varied across institutions, but some 
trends did emerge. 

Multiple respondents cited institutional or departmental policies that restricted the ability of 
undergraduates to pursue international opportunities. At the departmental level, these policies 
included such things as the structure and scheduling of the curriculum for undergraduate STEM major 
and/or limited flexibility for meeting course or credit requirements (6 responses): 

We actively promote these opportunities, but the structured curricula of many students in these 
disciplines make it difficult for them to take advantage of international opportunities. 
(Administrator at a 2007 PIPE awardee) 

...by instituting policies limiting the number of credit hours for a degree, STEM faculty have 
clamped down on both the number of elective credits and the number of required credits outside 
the faculty's immediate control. (Administrator at a 2007 PIPE awardee) 

One of these respondents also cited increasing pressure on undergraduates to graduate early, which, 
combined with departmental restrictions on course requirements and the schedule of course offerings, 
was seen as a barrier to international engagement: 

There [is] a push for decreasing graduation time and time spent abroad can add to the time taken 
to graduate. (Administrator at a 2012 PIPE awardee) 

Multiple respondents also cited the high cost of international experience and limited funding sources 
such as fellowships that might help offset these higher costs (5 responses). Even indirectly, the cost of 
an international experience can be high, if, for example, time abroad lengthens a student’s time to 
degree completion (thus requiring additional semesters of tuition and other costs of obtaining an 
undergraduate degree). Three respondents mentioned the challenges of international travel as a 
barrier, including such factors as export control regulations, or “excessive” university protocols for 
ensuring students’ safety while abroad. 
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5.2.3 Factors Supporting Graduate Students International Engagement 

Thirty-four respondents deseribed institutional policies or practices that facilitated graduate student 
engagement in international research and education. Eight respondents provided general assurances 
that their institutions supported international experiences for graduate students. For example: 

We encourage graduate students to participate in international activities and do what we can to 
make this as easy as possible for the students and their advisors. (Administrator at a 2012 PIRE 
awardee) 

Other administrators reported more specific mechanisms to support graduate students’ international 
engagement. Some cited existing partnerships with international institutions (7 responses) or the role 
of the institution’s study abroad or international programs office in assisting graduate students (5 
responses). Notably, when describing mechanisms of support for graduate students, fewer 
administrators mentioned their study abroad office than the number who mentioned this office when 
describing such support for undergraduate students. 

The institution has Memoranda of Understanding with institutions across the globe. These MOUs 
generally provide a mechanism for exchange of graduate students, post-docs, and faculty. 
Mechanisms will vary depending upon funding source. We currently have graduate students 
working in Europe, Africa, South America and Southeast Asia and probably other places of 
which / might not be aware. (Administrator at a 2010 PIRE awardee) 

The office that supports undergraduate study abroad can also support graduate study abroad. 
Eewer graduate students avail themselves of these activities largely because of cost or 
restrictions on their funding by the funding agencies, or because they simply prefer to stay in the 
area throughout their training. The award to us of the PIRE was really good in addressing some 
of these limitations. (Administrator at a 2007 PIRE awardee) 

Other administrators indicated that funding was available for graduate students wishing to pursue 
international research or education (5 responses), or that the institution and/or departments were 
flexible enough to accommodate international engagement by graduate students (5 responses), for 
example, by assisting with credit transfers. In addition, three administrators mentioned that there was 
strong support from individual faculty for graduate student engagement in international research and 
education; none of the respondents mentioned this factor when describing support mechanisms for 
undergraduate international engagement. 

5.2.4 Factors Hindering Graduate Students International Engagement 

Twelve administrators described institutional policies or practices that hindered graduate student 
engagement in international opportunities. Multiple respondents cited the high cost of international 
research or educational opportunities and inadequate availability of funding mechanisms (4 
responses). For example, one respondent reported that graduate students who traveled abroad risked 
losing the tuition waivers they received as part of the university’s financial support to graduate 
students: 

Since most graduate students are on assistantships and do not pay tuition, study abroad 
programs do not receive a "tuition return" for graduate students participating in international 
programs as they do for undergraduates. (Administrator at a 2007 PIRE awardee) 

Others mentioned barriers related to international travel (4 responses) including a risk-averse culture 
in which administrators were concerned with graduate students’ safety abroad, and export control 
regulations (e.g., on what equipment or data students could bring with them to support the research). 
International graduate students attending the institution on a single-entry visa had concerns about 
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their ability to re-enter the U.S. after taking advantage of an international research opportunity. Other 
respondents cited a lack of structures to facilitate graduate student travel abroad (2 responses). 

Many units encourage graduate students to engage abroad, and there are some good funding 
systems in place. But there are few resources to help them prepare to do so, or to provide pre- 
departure education around health, safety, and cultural adjustment. (Administrator at a 2007 
PIRE awardee) 

5.3 Changes Resulting from PIRE 

The majority of administrators indicated that their institution’s PIRE award had not resulted in any 
changes in policies or practices related to undergraduate or graduate education in STEM fields (32 
respondents). Among the 14 administrators who did cite a change resulting from PIRE, 1 1 noted a 
change affecting undergraduates and 14 cited a change affecting graduate students. 

5.3.1 Changes Affecting Undergraduates’ International Opportunities 

The most frequently mentioned change cited as affecting undergraduates’ international engagement 
was greater faculty awareness and improved promotion of international opportunities (5 responses). 

The greatest impact has been on undergraduate education: PIRE programs have made STEM 
faculty more sensitive to international collaborations and more knowledgeable about 
international partners. Thus, faculty are more likely to promote international study and research 
and better advise their students. (Administrator at a 2007 PIRE awardee ) 

Multiple respondents also noted that PIRE demonstrated the value of providing international research 
opportunities for undergraduates (3 responses), and, as a result, undergraduates were more likely to 
pursue research experience. 

[PIRE] demonstrated the need and value for research learning at the undergrad level. 
Encouraged undergrads to pursue graduate education and research careers. Demonstrated the 
value of experiential (versus strictly classroom) experience gained overseas. (Administrator from 
a 2005 PIRE awardee ) 

The participation of our undergraduate students in multi-disciplinary and internationally 
spanned collaborative research projects and their valuable contribution reflected in some have 
encouraged more involvement of our undergraduate in the research labs of our institution. 
(Administrator at a 2007 PIRE awardee) 

Three respondents cited the new efforts to support undergraduates conducting research overseas. 

We identified a research liaison within the Office of Research to support students studying abroad. 
We also developed closer cooperation with the International Student Office to help Pis organize 
housing, healthcare and other requirements for students working on research projects overseas. 
(Administrator at a 2010 PIRE awardee) 

5.3.2 Changes Affecting Graduate Students’ International Opportunities 

The most frequently mentioned change cited as affecting graduate students’ international engagement 
was that expanded awareness of the value of such opportunities led institutions to increase their 
support for such experiences (7 responses). 

The most profound [change] is the realization that graduate education, not only research, can be 
done effectively across borders, taking advantage of complementarity of expertise and facilities. 
(Administrator at a 2005 PIRE awardee) 
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We offer greater support and encouragement for graduate students to consider spending time 
abroad at collaborating institutions. (Administrator at a 2007 PIRE awardee) 

Two respondents indieated that the PIRE award had increased faculty awareness of the advancement 
of research excellence outside the U.S. (i.e., implying that graduate students were both the vector 
through which faculty gained this awareness and the likely heneficiary of this newly-engaged 
faculty). 

Faculty gained a greater realization of the competitive landscape in research/innovation among 
different regions of the world, and how other countries have taken a quantum leap in research 
and higher education. (Administrator at a 2005 PIRE awardee) 

After our graduate students got the opportunity to visit and work in top-ranked research facilities 
outside [the] U.S., they realized that some of the best research activities and outcome may be 
done outside [the] U.S. too. This have [sic] encouraged our research faculty members to keep 
themselves up to date with the latest progress in their field done in other countries, especially 
European, Asian, and South American countries. (Administrator at a 2007 PIRE awardee) 

5.4 Challenges 

Most institutional administrators (38) reported that their institution had encountered no challenges 
related to faculty or student activities in other countries. Among the seven who did descrihe 
challenges, two mentioned resource or funding issues: 

Not being able to use grant funds to foster international relationships with researchers in less 
developed countries has limited the choice of institutions with which we partner. Some 
institutional funds have been used for support of non-grant funded researchers, but not robustly. 
(Administrator at a 2009 PIRE awardee) 

Two mentioned inflexibility of program requirements for students in some disciplines: 

The only challenges had to do with inflexibility in some engineering programs that made it hard 
for students to go abroad. (Administrator at a 2007 PIRE awardee) 

And others mentioned safety (1), lack of administrative support (1), or intellectual property issues (1): 

Some of our major research partners were located in COUNTRY. During the first year of our 
PIRE, we sent three students to COUNTRY to work on a long- established collaborative research 
project. Our students faced some safety issues. Our institution placed a policy not to send any 
students to COUNTRY until the time that the issue with drug cartels are [sic] resolved. As the 
safety of our participants were [sic] our highest priority, even though we could make an attempt 
to waive this restriction, we decided not to do so. This hindered our research collaboration with 
COUNTRY. (Administrator at a 2007 PIRE awardee) 

We did not have a coordinated resource to help faculty work through the HR, legal and benefits 
questions to be able to send students abroad. As described earlier, we worked through this. 
(Administrator at a 2007 PIRE awardee) 

The ... universities [in COUNTRY] were not used to dealing with the [intellectual property] (IP) 
framework that American universities operate within. Our technology [sic] had a long discussion 
with the lead [COUNTRY] university, which came to a successful exchange agreement including 
IP terms. (Administrator at a 2005 PIRE awardee) 
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6. Societal Impacts 


All proposals submitted to NSF must demonstrate evidence of the potential for their project’s broader 
impacts. Given the international partnerships inherent to PIRE projects, NSF was particularly 
interested to learn whether or not PIRE projects have contributed to global societal challenges, 
loosely defined as problems facing human populations that any single nation was ill -equipped to 
tackle alone. In addition, for the 2012 PIRE projects, NSF’s solicitation limited proposals to those 
that were focused on the NSF-wide investment in Science, Engineering, and Education for 
Sustainability (SEES): 

A sustainable world is one in which human needs are met equitably without harm to the 
environment, and without sacrificing the ability of future generations to meet their needs. 

Meeting this formidable challenge requires a substantial increase in our understanding of the 
integrated system of society, the natural world, and the alterations humans bring to Earth. NSF's 
Science, Engineering, and Education for Sustainability (SEES) activities aim to address this need 
through support for interdisciplinary research and education in all fields of science and 
engineering. Especially encouraged is research on global sustainability issues including, but not 
limited to, climate change, clean energy, food security, biodiversity, and communication 
networks ” (NSF, 201 1). 

Altbougb this fourth round of PIRE awards directly focused on a set of specific global challenges, the 
evaluation probed senior investigators’ perceptions of PIRE’s contributions to society. Pis and FIs 
described wbat they perceived to be the benefits of the PIRE project for society at large, including 
those who had not participated directly in a PIRE project. 

This chapter presents findings related to perceived societal or broader impacts of tbe PIRE projects. 
Specifically, the findings address the following research question: 

• RQ 7. How have PIRE projects contributed to research that may inform global societal 
challenges? 

We summarize the perspectives of Pis and FIs on how their PIRE project benefited society at large 
and extended benefits beyond those directly participating in the project. Findings in this chapter come 
from PI and FI surveys and include descriptive summaries of their responses as well as specific 
examples provided by these senior investigators. Key findings include: 

• The three most frequent types of benefits of PIRE described by senior investigators were insights 
about the natural environment (43 percent of PI responses, 24 percent of FI responses); 
technological advances that would benefit society (21 percent of PI responses) and PIRE’s 
promotion of cross-cultural exchange (18 percent of PI responses, 32 percent of FI responses). 
Specific examples included scientific advances in sustainable energy, conservation, 
understanding climate change, and mitigating hazards from disasters. 

• Senior investigators (74 percent of Pis and 78 percent of FIs) noted that their PIRE project has 
advanced knowledge or led to discoveries that might help address global challenges in health, 
environmental and other areas in which solutions appear unlikely to emerge from the efforts of 
one nation acting alone. Asked how their PIRE project could help address these challenges, the 
mechanism that senior investigators most often mentioned was that PIRE projects leveraged the 
expertise, resources, and perspectives of a range of international researchers. 
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• Pis and FIs indicated that findings from their PIRE project could inform policy and planning, 
provide tools or technologies applicable to global challenges, and provide improved academic 
models for educating the next generation of researchers. 


6.1 Principal and Foreign Investigators’ Perception of the Benefits of their 
PIRE project to Society 

Prior to survey data collection, we conducted two focus groups with eight Pis each at the May 2014 
PIRE PI meeting to understand how Pis would interpret the “global societal impacts” of PIRE. 
Virtually all of the focus group participants singled out the educational opportunities for graduate and 
undergraduate students that their PIRE project (or the program as a whole) was providing. As a result, 
we tailored the PI and FI survey items to encourage respondents to think beyond the direct benefits to 
PIRE participants themselves. 

Exhibit 6. 1 summarizes the key themes that emerged from qualitative analyses of PI and FI survey 
responses. 


Exhibit 6.1 : Key Themes from Pi and Fi Reports of the Benefits of PiRE to Society 



Percent of Responses 

Benefit Reported 

Pis 

(N=158) 

FIs 

(N=78) 

Insights about the natural environment (and human interactions with this 
environment) 

43% 

24% 

Technological advances 

21% 

0% 

Promoting international collaboration and cross-cultural exchange 

18% 

32% 

Knowledge gained from cross-cultural comparisons/sharing of best practices 

6% 

12% 

Improved program models 

5% 

10% 

Advancing knowledge in a particular field or science in general 

4% 

22% 

Predicting and/or mitigating hazards 

4% 

3% 

Fostering global citizenship/public awareness 

4% 

5% 


Sources: Principal Investigator Survey, Item GI; Foreign Senior Investigator Survey, Item E4. 
Notes: 

Pis: N=158, missing=10. FIs: N=78, missing=44. 


Pis and FIs cited specific advances in research related to the human-natural environment, including 
research in sustainable energy, climate change, biodiversity, conservation and water resources or the 
impact of human populations on the natural environment (43 percent of PI responses; 24 percent of FI 
responses). 

The PIRE project has fostered collaboration across the Pan American region to measure the 
sustainability impacts of bioenergy development in areas of the natural environment and 
socioeconomic issues. These types of measurements are largely overlooked as the various 
countries in this region of the world ramp up biofuel production. (2012 PI) 

Our research will enhance [sic] quality of large wind farm designs of the future. This will enable 
larger penetration of this renewable energy source for our electricity grid. Such outcome [sic] 
would have transformative societal impact. (2012 PI) 

More than 20 percent of Pis’ responses cited potential benefits of technological advances resulting 
from their PIRE projects. 

Our research project is [creating] “meaning representations ” that can improve automatic 
machine translation and all forms of multi-lingual information processing. This could vastly 
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improve world-wide communication, lower language and cultural barriers and encourage 
cooperation and collaboration. (2005 PI) 

The coupling of technological solutions with social considerations at the outset. And looking at 
the types of technological solutions that might cause behavioral change in energy use. (2012 PI) 

Nearly one -fifth of Pis (18 percent of PI responses) and nearly one-third of FIs (32 percent of FI 
responses) cited the cross-cultural exchange as a key benefit of PIRE to society: respondents credited 
this type of collahoration with allowing researchers to better address global challenges. 

[The] PIRE project is a good example that people at different countries can work together to 
produce useful results. This give [s] [a] very good example that we can work together to resolve 
global issue and benefits to societies. (2007 PI) 

It set an example of outstanding and equal international collaboration (many other international 
projects had [PEOPLE IN THIS COUNTRY] only serve as guides or logistic people). Local 
people benefited from the interaction with American scientists and students. (2007 PI) 

It creates a critical mass of scholars who aim at advancing science regardless of National 
borders. (2012 PI) 

[ One of the benefits is the] diffusion of innovation and broader dissemination of knowledge and 
experience regarding water innovations. (2012 PL) 

The project shows how a multi-disciplinary and multi-cultural team of researchers can 
successfully work together on globally important scientific questions. This is a pretty unique 
situation worldwide. (2010 PI) 

Science is an international activity. It is best done by people working together based on their 
interests. Any program that supports this will help the societies involved. (2010 El) 

6.2 PIRE’s Contributions to Addressing Specific Global Challenges 

The National Academies of Science and Engineering (with their partners, the Institute of Medicine 
and the National Research Council) have identified a set of “global challenges” out of recognition that 
“Many of the world’s greatest health, environmental, and security threats are beyond the ability of 
any one nation to confront by itself’ (National Academy of Sciences, 2015). Examples of areas where 
the Academies indicated that internationally coordinated responses were needed included: 

• Meeting increased global demand for energy without causing irrevocable harm to the 
environment, reducing the human forcing of climate change, and developing and implementing 
economically viable low-carbon energy technologies; 

• Improving access to clean water that is threatened by rapid human population growth, increasing 
urbanization and poor water management; 

• Addressing reduced agricultural output and increased food insecurity due to poor soil, lack of 
high quality seeds, and inefficient use of water or fuel; 

• Minimizing risks of dual-use technologies in the life and nuclear sciences that can be used to 
improve lives but also to manufacture weapons; 

• Promoting and protecting innovation with appropriate protection that does not stifle further 
advances; 
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• Building resilience to hazards from natural and technological disasters; and 

• Addressing demographic challenges from population growth, aging, and urban population 
dynamics in developing countries. 

Nearly three-fourths of Pis and more than three-fourths of FIs noted that their PIRE project has 
advanced knowledge or led to discoveries that might help address global challenges in areas in which 
solutions are unlikely to emerge from the efforts of one nation acting alone (Exhibit 6.2). 


Exhibit 6.2: Percent of Principai and Foreign investigators Reporting that Their PiRE Project 
Has Heiped Address Giobai Societai Chaiienges 



Pis 

FIs 

My PIRE project has advanced knowledge or led to discoveries that 
might help address global challenges such as those listed above, or In 
another area In which solutions are unlikely to emerge from the efforts 
of one nation acting alone 

74% 

78% 


Sources: Principal Investigator Survey, Item G3a; Foreign Senior Investigator Survey, Item E6a. 
Notes: 

Pis: N= 166, 4 missing. FIs: N=lll, 11 missing 


When asked to elaborate on how their PIRE project would contribute to addressing these challenges, 
a common theme among both Pis (25 percent of PI responses) and Els (35 percent of FI responses) 
was that the PIRE project leveraged the expertise, resources, and perspectives of a range of 
international researchers.'^ 

Understanding the nature of climate/greenhouse gas interactions is paramount and extremely 
difficult. It is only by forging the expertise from numerous nations that we move the science 
forward. (2010 PI) 

Our PIRE project addresses materials under extreme conditions for nuclear systems. Leveraging 
unique facilities and talents at different institutions will advance knowledge and address global 
challenges in this field. (2012 PI) 

Understanding how humans understand speech and language is best done by studying this human 
facility across multiple languages — to discover language universals — and requires perspectives 
from multiple disciplines. No single nation has the human capital to cover all the necessary 
expertise, often due to historical reasons. Rather than trying to develop the expertise in one 
nation, it is infinitely more effective to collaborate. (2005 PI) 

[ COUNTRY] part is good at the materials synthesis, U.S. part is good at the data collecting with 
the advanced instrument and the data analysis. (2010 FI) 

The problem of water supply quality and quantity is highly varied across the globe. The 
international perspective and expertise achieves far more than the efforts of one country alone. 
(2012 FI) 

The issues dealt with in this project are global in the sense that climate and lands changes are 
happening everywhere, however, the way to understand and manage these problems need [sic] 
regional perspectives that lie beyond the borders of the U.S. The international engagement 
process is the only way to truly address environmental issues that impact different 
regions/peoples in different ways. (2010 FI) 


1 15 Pis and 63 FIs gave responses (PI survey. Item G3b; FI survey Item E6b) 
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By adopting a comparative perspective across 6 countries, the PIRE project has made it possible 
to identify global patterns, problems and potential solutions. (2012 FI) 

Other responses cited the multi-national applicahility of the PIRE project’s findings (10 percent of PI 
responses, 19 percent of FI responses), and/or the potential benefits of leveraging PIRE to huild a 
cross-national consensus on the best possible solutions (10 percent of PI responses). 

Unmet global electricity demand, in particular, has significant potential in terms of increasing/ 
reducing greenhouse gas emissions. Lessons learned in the U.S. & [COUNTRY] should be 
transferable to other parts of the world. (2012 PI) 

The project is addressing many questions related to sustainability of bioenergy in various... 
countries [in this GEOGRAPHIC REGION]. This multidisciplinary ... experience is a large effort 
that cannot compare to any effort from a country. Data will be comparable among dissimilar 
bioenergy feedstocks, nations, climates, cultures and socioeconomic realities. (2012 FT) 

Results from our PIRE has led to consensus amongst world leaders in fire science — consensus 
that we must recognize the changing role of disturbance globally and how this relates to resource 
use and needs of all humans. This consensus should help guide international 
cooperation/policies. (2010 PI) 

By demonstrating the obstacles, impediments, as well as opportunities to be recognized in making 
innovations both useful and adaptable cross-nationally, and in different regulatory and policy 
contexts. (2012 PI) 

FIs in particular also focused on the positive impact the project had on education and opportunities 
for students (35 percent of responses), including showing them the value of international 
collaboration (13 percent of responses), which they believe will create an academic environment 
more suited to addressing global challenges. 

More awareness among the graduate student of the quality of research, work ethics, and overall 
ability to think differently. (2007 FI) 

Although only several undergraduate and graduate students were involved in the field and lab 
work related to the PIRE project, they have become the "seed” to intensify the interaction of 
geoscientists from [ COUNTRY] and the U.S. in the near future. They have learnt a lot and 
developed friendship with American professors and students. Some of them have published 
papers in peer-reviewed international journals and became independent researchers. (2010 FI) 

6.3 Implementation of Research Findings 

Senior investigators also provided insight into ways that society could apply the results of PIRE to 
address challenges of global significance or scale. The most common mechanism for implementing 
PIRE results was by informing policy and planning (21 percent of PI responses and 14 percent of FI 
responses). 

The publications from the PIRE project offer many suggestions for changes in educational 
programs, policies, and practice that would increase educational opportunities for the children of 
immigrants. (2005 PI) 


150 Pis and 21 FIs described ways that society could implement PIRE results (PI survey, Item G2; FI 
survey, Item E5). 
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Our contributions to the understanding of marine connectivity are being used to assist in the 
development of marine conservation planning in the PIRE region and other areas globally. (2007 
PI) 

This project is focused on sustainability along many dimensions, which is something that policy 
makers across the globe are grappling with. This project should provide fundamental information 
that contributes to these real world problems. (2012 PI) 

Under [PENDING LEGISLATION] , it may become possible to finance pilot projects in order to 
determine broader feasibility of renewable microgrids. (2012 PI) 

Cities are considering the complexity and large supply chains of infrastructure, cities are using 
our methods for tracking GHG emissions, cities are interested in metrics we have developed to 
compare efficiency. In [the] future we hope to address the solutions and how we can best design 
combined social-infrastructural solutions to address sustainability and health goals of cities. 
(2012 PI) 

V7e produced detailed reports for each [GEOGRAPHICAL UNIT WITHIN A COUNTRY] and 
handed those to several government agencies that issue permits for commercial extractions and 
biological collecting. We also invited main NGOs to [sic] and orally presented the general 
results and gave copies of the reports (also available online). Results produced on the ...fauna, 
are now being used as the basis for more integrative and complex analyses related to global 
climate change. (2005 PI) 

I anticipate that the growing body of knowledge arising from the PIRE project will inform 
decisions about climate impacts on population dynamics of wild birds, and improve 
understanding of the main drivers. In turn, this will yield important insights into program and 
policy alternatives that could mitigate such effects. (2007 PI) 

Another 10 percent of PI responses (and one FI response) proposed that PIRE results could 
specifically aid in planning and monitoring to mitigate the dangers of natural disasters: 

Society can implement dam and levee monitoring systems and can convey the information much 
like weathercasting is being used today in order to inform people so they are better prepared in 
the event of a disaster. (2012 PI) 

The project facilitated the development of novel volcano monitoring solutions, which have 
already been tested in harsh [REGIONAL] weather conditions and will work equally well across 
the world. The [INSTITUTION] for example, is adopting the "multiparameter instrumentation" 
developed with the support of our PIRE project. (2005 PI) 

Thirteen percent of PI responses and 10 percent of FI responses suggested that PIRE findings could 
he used hy industries and other stakeholders to address global challenges: 

Industry is using our academic results to develop better machine translation, in more languages 
(including disadvantaged and low-resource languages), and for more applications beyond 
defense and commercial. (2005 PI) 

In order to take advantage of what we learned, and produce devices or materials with 
advantageous properties, it will be necessary that private companies get involved. They are the 
link where prototypes of potential devices could be eventually fabricated in large volumes. Tor 
this to happen, companies should receive some type of incentive to approach researchers at 
universities to make the initial investments. (2007 PI) 
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Baseline information on earth structure obtained from this project is being used by mineral and 
oil exploration companies to improve exploration models. This sensitive information is also being 
used to improve estimates of seismic hazards and risk. (2005 PI) 

Some of our ... research outcomes have been used by [COMPANY] and [INSTITUTION] to scale 
weather research forecasting to clusters of computers across the Atlantic Ocean. (2007 PI) 

Some experts at nuclear power plants [in] [ COUNTRY] participated for discussion and then they 
have been trying to implement our research findings. (2007 FI) 

Seventeen percent of PI responses suggested that PIRE had (or would) lead to improved tools and 
technologies useful for addressing global challenges: 

The method of energy conversion is very promising and could provide an important clean energy 
source for the important small temperature difference regime. (2010 PI) 

New bio-based materials could form sustainable feedstocks of materials for energy and other 
engineering applications. This is aimed at more environmentally friendly technology. (2012 PI) 

Respondents also said that PIRE findings could address global challenges by informing future 
research (8 percent of PI responses, 10 percent of PI responses) and improving academic program 
models (5 percent of PI responses, 10 percent of PI responses). 

Increasing the number of scientists with expertise in managing and analyzing very large datasets 
is also vital to the future of our nation. Since this transformative technology is broadly applicable 
to any scientific project struggling to manage and analyze the volume of data produced, the 
[PIRE PROJECT] and its facilitative impacts are likely to persist long after the PIRE project has 
ended. (2010 PI) 

The program has become a nationally recognized model for designing and managing an 
international REU. It is a model for increasing the number of STEM majors participating in 
research abroad. (2005 PI) 

The new method of teaching with live, video, simulcast of lectures given by top leaders of 
scientific research will be followed by educators and researchers in many other fields, and 
probably become a standard method for higher education. (2009 PI) 

[The project is] Improving the training of young scientists in the service of the [PEOPLE IN 
THIS COUNTRY] and international society. Generating valuable knowledge for biodiversity 
conservation of southern [GEOGRAPHICAL REGION]. (2005 El) 
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Since its inception in 2005, the PIRE program has funded projects across a hroad array of scientific 
disciplines, in an effort to catalyze long-term, sustainable international partnerships for research and 
education. NSF deliberately restricts PIRE to intellectually substantive collaborations in which the 
international partnership is essential to the research effort. Moreover, the program includes a strong 
emphasis on providing early career opportunities for postdoctoral, graduate and undergraduate 
students to participate in meaningful international research and education. 

The primary goal of this first evaluation of PIRE was to assess how well PIRE is meeting its key 
objectives: to foster meaningful and sustained collaborations between U.S. and foreign partners; to 
promote research excellence — in particular, advances in knowledge and discovery that could not 
occur without the combined expertise and resources of an international team; and to cultivate 
increased global engagement among the next generation of scientists and engineers. The evaluation 
was also designed to document the outcomes and experiences of PIRE participants, to explore PIRE’s 
contributions to addressing societal challenges, and to examine institutional policies and practices that 
might promote or impede faculty or student international engagement. 

Importantly, the evaluation was aimed at investigating the value-added benefits of the PIRE program 
to fostering international collaboration and producing high quality research. Across NSF’s many 
programs, PIRE is distinctive in that it requires an international collaboration, so a key question is 
what benefits (or unintended consequences) result from this requirement. To this end, the evaluation 
compared the outcomes of the first 59 PIRE awards to those of other NSE research grants with 
characteristics similar to these PIRE awards. Primary outcomes for which we compared PIRE to other 
NSE grants included: the quantity and quality of research produced; the level of foreign engagement 
in this research; and the nature and sustainability of international collaborations during and 
subsequent to the award period. The evaluation also explored participant experiences related to the 
grant, perceptions of the effects of PIRE, including senior investigators’ perceptions of PIRE’s 
contributions to addressing societal challenges, and examined institutional policies and practices for 
supporting undergraduate and graduate students’ international engagement. 

PIRE produces strong research outcomes. 

The evaluation found evidence that PIRE is furthering its objectives with no detectable tradeoff costs 
to research productivity or quality. Research outcomes of PIRE projects and participants compare 
well to those of similar NSE-funded grants without the explicit requirement for international 
collaboration. Further, the evaluation findings underscore the importance of explicitly supporting 
foreign institutional (and investigator) involvement in order to promote international collaborations 
that can seed productive and impactful activities. While there were no statistically significant 
differences between PIRE and the comparison group in the quantity or quality of research produced, 
PIRE publications had a significantly higher proportion of foreign contribution. Moreover, there is 
evidence that PIRE is playing an important role in the research productivity of early -career 
investigators. At the participant level, PIRE postdocs, on average, had a larger number of post-onset 
annual publications and these publications had a larger impact compared to their peers working on 
other NSF grants; PIRE graduate students also had a higher average number of annual post-award 
publications. 

PIRE fosters meaningful, sustainable international collaborations. 

PIRE Pis, postdocs and graduate students were more likely than their counterparts in other NSF 
projects to report collaborating with a foreign investigator during the project. More importantly. 
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among participants who collaborated with a foreign colleague during their participation in the project, 
PIRE participants were more likely than their peers to continue collaborating with foreign 
investigators. This finding provides important evidence that the PIRE program is uniquely structured 
to foster meaningful international collaborations that are sustainable beyond the life of a single grant. 

Given research showing that internationally coauthored publications tend to have higher citation 
impacts (Adams, 2013), we might have hypothesized that the PIRE program would result in higher 
quality research, but, due to the complexity of PIRE’ s multi -investigator, multi-institutional, and 
multi-national collaboration, it seemed equally plausible that, at least in the short term, PIRE could 
have resulted in lower levels of productivity or quality, if these were the tradeoffs of administering a 
PIRE award. The findings of this evaluation address these competing hypotheses: PIRE participants 
are more likely than participants in other NSE-funded projects to establish and maintain 
collaborations with foreign colleagues, and attaining this objective does not come with a cost to 
research productivity or impact, even when compared to the high standards set by NSE-funded 
projects. 

U.S. institutions value PIRE and are working to promote international engagement. 

Most administrators reported that their U.S. institutions actively support international engagement by 
faculty and students by helping with arrangements for research or education abroad, coordinating 
with a partner institution in another country, and providing faculty with grant-writing assistance, or 
even seed money for planning such endeavors. Many of these institutions reported that the PIRE 
award aligned with explicit institutional goals to raise their own global profile or build international 
partnerships. Eor undergraduate and graduate students pursuing research or educational activities 
abroad, institutional administrators cited the importance of their campus study abroad or international 
programs office and faculty support. Administrators described a greater number of barriers to 
undergraduate than graduate student participation in international opportunities. Eor undergraduates 
majoring in an S&E field, administrators cited departmental curricular requirements and the high cost 
of educational activities abroad as deterrents. Eewer barriers were cited as impeding graduate 
students’ international engagement, but some administrators noted that such students risked losing 
tuition waivers and international students on single -entry visas were reluctant to leave the U.S. 

Despite these barriers, institutions also reported that the PIRE award had raised awareness and 
demonstrated the value of having students involved in international research and education, leading to 
increased support for providing such opportunities. 

PIRE participants cite multiple benefits of the program. 

Participants valued their PIRE experience. Nearly all postdoctoral, graduate and undergraduate survey 
respondents would encourage their peers to participate in research and education opportunities 
abroad, and the vast majority (more than 90 percent) indicated that different cultural perspectives 
improved the quality of the research, and that the research benefitted from access to foreign expertise 
and data acquired in foreign locations. 

Senior investigators likewise cited benefits from leveraging the expertise, resources, and perspectives 
of a range of international researchers, and also reported that PIRE is producing broader societal 
benefits. Both U.S. -based Pis and foreign senior investigators reported that their PIRE project had 
advanced knowledge or led to discoveries that might help address global challenges in areas in which 
solutions are unlikely to emerge from the efforts of one nation acting alone. Pis and Els indicated that 
findings from their PIRE project could inform policy and planning and provide improved 
technologies in such diverse areas as marine conservation, renewable energy, and predicting and 
mitigating the effects of natural disasters. 
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Based on the evaluation findings, we propose four recommendations for NSF to consider as it looks 
ahead to future cohorts of PIRE awards and to the next evaluation of the program. 

Collect information about PIRE (and other program) participants more systematically 

Given the challenges of locating contact information for PIRE and comparison project participants 
prior to collecting data for the evaluation, we recommend that NSE consider developing and 
implementing a process to collect information from all participants in activities funded hy NSE 
awards. Although Pis list the names of project participants in annual reports to NSF, this practice 
does not always yield accurate or complete information needed for NSF to evaluate its programs. 
Requiring Pis to report more detailed information on participants seems potentially burdensome, 
given the information NSF already requests. Instead, we propose that Pis distribute a short 
questionnaire to each participant, pre -labeled with the NSF award ID (whether or not a participant 
received funding directly from that award), that would ask participants to provide the following 
optional information (ideally, online): 

• Residential address; 

• Institutional and permanent email addresses; 

• Date participation began; 

• Highest degree earned at start of participation; 

• Degree being pursued at start of participation (if applicable); 

• Major field for degree or primary field of research at start of participation; and 

• Gender, race ethnicity, and disability status. 

In addition, NSF could consider notifying participants that they may be invited to participate in a 
future study about their experiences and asking the participant to specify how they prefer to be 
contacted. Given the ubiquity of mobile and smartphones, another possibility is to request a cellphone 
number and opt-in to receive news of any future study via text message to that number. 

Had we begun the study with a better record of participant information, the evaluation would likely 
have achieved higher response rates. In addition, the information on participants who chose not to 
respond to a survey invitation would have allowed us to weight the survey data for non-response, thus 
mitigating the possibility of non-response bias in the survey data. 

Consider mechanisms to alleviate administrative burden on PIRE Pis 

Anecdotal reports from PIRE Pis and NSE suggest that PIRE awards impose a particularly severe 
administrative burden on lead Pis and some co-PIs, due to the increased logistical planning that 
international travel and collaborations require. Although the data collected for this evaluation do not 
confirm or disconfirm any hypotheses about administrative burden of PIRE, it is noteworthy that 
PIRE positively affected the research productivity of both postdoctoral researchers and graduate 
students, but did not affect the productivity of Pis (Exhibit 3. 1 1). NSF might consider requiring PIRE 
projects to fund a project administrator to help manage the projects and thus allow the Pis to focus on 
the research agenda of the award. 


Abt Associates 


Evaluation of NSF’s PIRE program: Volume 1 1 pg. 79 




Recommendations 


Facilitate funding and support for foreign investigators participating in FIRE 

Nearly one -quarter (24 percent) of foreign senior investigators reported that they had received no 
funding from any source (including a governmental agency or their own institution) for their 
participation in PIRE. Very few reported that one of the counterpart agencies NSF has identified in 
the United Kingdom, Japan, Russia, or the Americas (including North, South and Central America 
and selected Caribbean nations) had provided funding (Exhibit 4.39); the largest share of FIs who did 
report funding for PIRE indicated that funding came from their institution. When Pis were asked to 
identify challenges to participating in PIRE, seven of twelve responses mentioned a lack of funding or 
funding insufficient in light of their contributions to the project. In light of this finding, we 
recommend that NSF consider exploring new agreements and/or broadening its partnerships with 
research agencies in other countries to enhance opportunities for foreign partners on PIRE projects to 
obtain funding. 

Identify best practices at U.S. institutions for minimizing barriers to student research abroad 

Although most institutional administrators were, overall, positive about the opportunities for 
international research and education that the PIRE award provided students, some did cite barriers to 
student international engagement. For undergraduates, one barrier cited in particular for those 
majoring in science or engineering was that the structured nature of the undergraduate curriculum in 
these disciplines makes it difficult for students to miss a semester’s coursework on campus. For 
graduate students, there appeared to be fewer formal institutional resources or structures to provide 
pre -departure assistance with travel requirements (health insurance, safety considerations) or cultural 
adjustment; some institutions relied on their undergraduate study abroad offices to assist graduate 
students traveling abroad. 

We recommend that NSF consider identifying and disseminating “best practices” at institutions that 
are successfully sending larger numbers of S&E undergraduate and/or graduate students abroad for 
research. If some institutions have successfully made accommodations to allow undergraduates 
majoring in S&E fields to do research abroad without sacrificing completion of major curricular 
requirements, other institutions might benefit from knowing how these accommodations were made. 
Likewise, for graduate students, some institutions may have developed resources that focus 
specifically on the needs of graduate students traveling abroad (rather than adapting resources 
targeting undergraduates pursuing study abroad). Sharing these innovations across institutions could 
facilitate greater student global engagement than leaving institutions to invent processes for 
themselves. Where students have gone abroad for PIRE, institutional administrators point out that this 
increases faculty appreciation for the value of these experiences. Tapping faculty enthusiasm for 
student international experience may be one way to help foster inter-institutional sharing of these best 
practices. 
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Continuing grant. An NSF award in which a full amount is authorized hut released in annual suh- 
amounts pending satisfactory progress. Differs from a standard grant in which all funds are released 
at once at the start of the award. 

FI = Foreign senior investigator. A non-U.S. -based participant in an NSF award who is equivalent 
to a co-principal investigator in the U.S. 

Field normalized citation impact (field NCI). The ratio of the number of citations of the target 
paper to the “world average baseline” which Thomson Reuters defines as the average number of 
citations per paper for all reviews or research articles published in the same year as the target paper in 
a journal within the same journal category as the journal in which the target article was published. 
Information about journal categories can be found here: 
http.V/ip-science. thomsonreuters. com/mil/scope/scope scieMAA and here: 
http://wokinfo.com/essays/iournal-selection-process/ 

Journal normalized citation impact (journal NCI). Also known in some literatures as the ratio of 
observed to expected citations, is the ratio of number of citations of the target paper to the average 
number of citations per article for all reviews and research articles published in the same year and in 
the same journal as the target paper. 

Percentage of foreign authors per article is the ratio of the number of authors based at non-U.S. 
institutions to the number of unique authors of the article. For example, for an article with 10 co- 
authors, 4 of whom are based in two institutions outside the U.S. and 6 of whom are based at three 
institutions within the U.S., the article’s percentage of foreign authors is 40 percent. This indicator is 
available only for articles published in 2009 or later. For articles published earlier than 2009, 
Thomson Reuters cannot disambiguate multiple authors at the same institution. Instead, the 
percentage of institutions listed in author information that are outside the U.S. is used as a proxy for 
the relative contribution of non-U.S. co-authors to the article. See the definition for an article’s 
percentage of foreign institutions, below. 

Percentage of foreign institutions per article is the ratio of the number of unique non-U.S. 
institutions out of the number of unique institutions listed in author information per article. Because 
Thomson Reuters cannot disambiguate multiple authors at the same institution for articles published 
earlier than 2009, the ratio of non-U.S. institutions is used as a proxy for the relative contribution of 
non-U.S. researchers to the article. For example, a paper with 10 co-authors based in two institutions 
outside the U.S. and three institutions within the U.S., the article’s percentage of non-U.S. -based 
institutions is 40 percent (two out of five institutions total are non-U.S.). 

Percentage of foreign institutions per article for articles citing PIRE/comparison publications. 

Across papers citing one or more PIRE or comparison project-produced publications, the percentage 
of authors’ institutions per citing publication that were located outside the U.S.. 

Percentage of foreign institutions per article for articles cited by PIRE/comparison 
publication(s) Across papers that were cited by one or more PIRE or comparison project-produced 
publications, the percentage of authors’ institutions per cited publication that were located outside the 
U.S.. 

PI = Principal investigator. A senior-level, U.S. -based researcher on an NSE award, either the lead 
principal investigator or a co-principal investigator. 
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Pre-onset. The period of time prior to a PIRE or comparison project participant’s first year of 
participation in the project. 

Post-onset. The period of time including and after a PIRE or comparison project participant’s first 
year of participation in the project. 

S&E. Science and engineering. Used to refer collectively to the scientific or engineering fields 
typically supported by NSF. 

STEM: Science, technology, engineering and mathematics. Similar to S&E, this acronym is used in 
the literature to refer collectively to the set of fields in these four major areas. Sometimes it includes 
fields not directly supported by NSE (e.g., biomedical fields). 

Sponsoring institution. The home institution of the lead principal investigator on an NSF award. 
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